In this post, I will show you step by step how to integrate your ACI Physical Fabric with AWS ACI Cloud Fabric.  I would suggest first reading through the introduction section of “ACI/Cloud Extension Usage Primer (AWS) – Cloud Tenant Only (deploying Application Load Balancer with Service Graph on AWS)“.   That will give you a very good understanding of the components used for ACI Cloud Integration.

Before we start, let’s quickly discuss the main idea of how to set up the connectivity.    Let’s take a quick look at what you would do for connectivity if you had 2 ACI Physical Fabrics that you would put together to create a ACI Multisite fabric.    This is pretty straight forward.  To make the diagram less messy and to convey the idea,  I only have 1 APIC, 1 Spine and 2 leaves in each fabric.  For a production size fabric with multiple spines / APIC clusters and MSO clusters the idea is the same.

All of you can relate to the above easily and most of you reading this have probably already implemented this.  Most people need a DCI solution and ACI Multisite gives that to you very easily.   The main points to note here are that:

• Spines of each site connect to their  ISNs and run OSPF between Spines and ISN
• ISNs connect to the IPv4 cloud and you need to make sure that spine to spine routing connectivity is good
• MSO cluster should be able to reach both Site’s APICs OOB
• Once you setup Multisite through MSO, you will have BGP eVPN peering between spines of different sites.

When Setting up one or more Physical ACI Fabric to integrate with Cloud ACI Fabric, the idea is exactly the same.  However since you are now connecting over the public Ipv4 Network, you have to give the physical connectivity a little bit more thought to logically achieve the same sort of topology.   Let’s take a look at the figure below

You can easily see that the connectivity is very similar.   Some things to note here are:

• On the AWS side, you don’t have real Spines and Leaves and APICs.   You have a couple of Cloud virtual CSRs and a Cloud APIC also known as cAPIC.  You could think of the AWS side CSRs as taking the place of the Spines on the cloud side.  All of those components will need to have Internet connectivity and this will be done by using AWS Elastic IPs.  (Elastic IPs in AWS are nailed down public IPs that will not change when you reboot your ec2 instances (or any other instances that you run on AWS)
• We are connecting the physical fabric to the Cloud fabric over the public Internet (generally unless using AWS direct connect). So, it’s common sense that we will want to front end the fabric with a Firewall (cluster).
• The firewall needs to be configured to do NAT/PAT translations so that outside public addresses gets translated to inside addresses.   You specifically need to do NAT tranalation, IP to IP for APIC, MSO and anything else you may want to access directly from the outside like jump boxes, etc, etc.
• Behind the firewall cluster you would have a couple of vCSRs running on a couple of hypervisors (for redundancy of course). 
• What happens is that we bring up a full mesh of IPSec tunnels between the vCSRs on the physical side and the Cloud vCSRs on the AWS side.  
• The IPSec tunnels act as point to point links between the AWS Cloud CSRs and the physical Fabric side Cloud virtual CSRs.  
• Now your OSPF peering is able to form neighbor relationships all the way from cloud CSRs to physical Fabric vCSRs through the IPSec tunnels.
• Now for control plane BGP eVPN peerings are built from the physical spines to the cloud CSRs on AWS side.
• for data plane the vxlan tunnels are built from physical spine to cloud CSRs

I’m hoping that you notice that the physical connectivity is very similar with some minor additions.   Now,  if you think about this a little bit more, you have to do your cabling in such a way that the virtual CSRs on the physical side is in front of the ISN routers and behind the Firewalls.

Below is a simple way of achieving this.

 

The diagram is pretty self explanatory.   Some items I want to point out are:

• we make 2 VRFs in the ISN routers 
• if you study the diagram you will notice that vmware DVS port group DMZ-ISN has all the virtual components (like vCSRs, MSO, jumpboxes, etc, etc) that need to reach out to the internet / AWS fabric components
• The Spines connect to the ISN in the VRF Spine defined on the ISN.
• However for the Spine to reach out to the Internet, it first has to get routed through the virtual CSRs, because of the way the VRFs segregate the ISNs.   Recall that the virtual CSRs will have ipSEC tunnels going to the Cloud CSRs.
• MSO can reach both APICs (physical APIC and cAPIC in AWS (through the internet).  Note, that you could also place the MSO on the AWS cloud itself running on the virtual Service Engine (found through AWS console in AWS market place).  If you place the MSO on the physical Fabric side, like shown in this diagram you also have the option of running the MSO on the virtual Service Engine )based on Kubernetes Cluster).   If interested please read Deploying MSO on Cisco Application Service Engine (OVA based SE).

Once you are done cabling up like shown in the above diagram, your logical diagram on the on premise (physical side) will look like the below.

In the above logical diagram, the 2 ISNs are the same ISN but different VRFs in that ISN.   Separating out this way  makes it’s easy to visualize the connectivity. 

On the vCSR’s please make sure:

• to point the default gateway to the SVI of VRF ISN-DMZ,  (10.1.100.1) in my example.  
• On the VRF -DMZ please make sure to point the default gateway to the Connected ASA interface 10.1.1.1/30 in my case. 
• You could try to ping some destination on the Internet from the vCSR like 8.8.8.8 to make sure it’s working (given that your ASA is configured correctly for NAT and icmp inspection).

When you install the vCSRs on the on Prem Fabric side, please make sure that you license for the security feature of virtual CSR.   That has the IPSec capability and you will obviously need to bring up ipSec Tunnels from your virtual CSR.   Please See the Licensing CSR1000V guide on CCO.

Below is a excerpt from that licensing document.

Note:   During the install of the vCSR OVA you will be asked to choose the feature set you want.  If you happen to not have chosen “Security” or “AX” based feature set, you could install the security feature set after booting up the vCSR by executing the command “license boot level security” from the config prompt.   You will then need to reboot the vCSR for it to take effect.

The next step will be to configure the Spine interface facing the ISN and bring up OSPF on that interface.   You will have to configure the following:

First make sure that you configure your Fabric Access Policies and tie the interface on the spine (going to ISN) to a vlan pool containing vlan 4.

If the physical fabric is already part of a Multisite fabric, this has already been done.

Next using MSO add in the Physical Site

If the physical fabric is already part of a Multisite fabric, this has already been done.

Next Fill in the details in the form

If the physical fabric is already part of a Multisite fabric, this has already been done.

Next Click on Configure Infra

If the physical fabric is already part of a Multisite fabric, this has already been done.

Next, click on the Site just Added

If the physical fabric is already part of a Multisite fabric, this has already been done.

Next, fill in the information as requested:

If the physical fabric is already part of a Multisite fabric, this has already been done.

1. Click on the main Site Name
2. Turn on MultiSite
3. Put in Site ID (1 in my case)
4. Put in Overlay Multicast TEP ( this is the Head End Replication Unicast IP)
5. Put in BGP AS
6. Put in OSPF Info

Next Click on POD # and fill in required info

If the physical fabric is already part of a Multisite fabric, this has already been done.

1. click on POD #
2. Put in the Overlay TEP  100.100.100.120  in my case.  This is the Unicast Any Cast TEP IP.  BGP EVPN next hop will be set to this for the POD

Next, Click on Spines (one at a time) and fill in Required Information

If the physical fabric is already part of a Multisite fabric, this has already been done.

1. Click on Spines (one at a time)
2. Put in Interface #, IP and MTU
3. Turn On BGP Peering
4. BGP EVPN Router ID.  This will be a loopback on the Spine and is also known as CP TEP (control Plane TEP).  BGP will use this loopback to source the peering
5. Make Spine Route Reflector
6. Click Deploy

Next, go to the Physical APIC, Navigate to Infra Tenant, Fabric ext Connection Policies:

1. Put in the Extended Community, in my case extended:as2-nn4:5:16
2. Verify that POD ID  and Unicast Data Plane TEP is showing up there

Now, you are done configuring the spine Interface side.  Next, configure the ISN Interface connectint to spine and put it in matching OSPF config on that side.

On the ISN Router, check to make sure that the OSPF Neighbor to Spine has come up

Now, extend OSPF all the way to the vCSR Interfaces

Configure and check for neighbors on both vCSR Routers

One more Check you may want to do:

Please verify that the ASA firewall is allowing the following:

• tcp port 22 for ssh
• tcp port 443 for ssl
• udp port 500 and udp port 4500 for IPSec tunnels
• If you are using radius in DMZ for authentication then you will need to have some UDP ports opened up for radius use.  Default ports for radius are udp 1812/1813, but you could translate that as needed.

You can use different methods to do this verification.  What I like to do for fast check,  is:

• for tcp 22,  use netcat or just ssh into the ubuntu box from outside
• for tcp, 443, use netcat or just bring up a docker container with nginx and browse to it
• If your DMZ firewall is an ASA, there are some easy ways to check what ports are exposed all the way to the ASA.

Below I show you some steps for quick verification on ASA.

• Let’s say you have done some translation for a Ubuntu jumpbox, from outside to inside.  You have verified that you can ssh to the jumpbox from the Internet or your corporate network (depending on your access-list setup on the ASA)
• Let’s say outside address of the Ubuntu Jumpbox is 178.178.178.71
• You want to verify that the corporate firewall is passing ports UDP 1812 and tcp 1812 all the way to your DMZ ASA permeter firewall.

Example: Checking for UDP 1812 to see if it comes all the way to ASA Permeter Firewall through corporate Firewall

• • Step 1:
    • Spin up aws EC2 with Internet Connectivity.   Look for the public IP of the EC2 instance.   Let’s say it is 3.84.239.77
  • Step 2:
    • on ASA, configure the folllowing access-list:
      • access-list cap extended permit ip host 3.84.239.77 any4
    • start a capture on the ASA with the following command:
      • capture c access-list cap interface outside real-time ( assuming your outside interface name is outside)
  • Step 3:
    • ssh to EC2 and do the following:
      • dig  @178.178.178.71 -p 1812 cnn.com
  • Step 4:
    • Watch the ASA console to see if you see any hits on that.  Example shown below

Example: Checking for tcp 1812 to see if it comes all the way to ASA Permeter Firewall through corporate Firewall (steps 1 and 2 are the same as above, so if you did the above, you can go directly to step 3)

• Step 1:
  • Spin up aws EC2 with Internet Connectivity.   Look for the public IP of the EC2 instance.   Let’s say it is 3.84.239.77
• Step 2:
  • on ASA, configure the folllowing access-list:
    • access-list cap extended permit ip host 3.84.239.77 any4
  • start a capture on the ASA with the following command:
    • capture c access-list cap interface outside real-time ( assuming your outside interface name is outside)
• Step 3:
  • ssh to EC2 and do the following:
    • sudo yum install telnet -y
    • telnet 178.178.178.71 1812
• Step 4:
  • Watch the ASA console to see if you see any hits on that.  Example shown below. 

•  

O.K. now, all the Physical Side Prep work is done !

Updates:

• 6/1/2020:  AWS cAPIC release 5.0.1k is now out in AWS Marketplace.   Please install this release.
  • The following features are available in release 5.0.1k
    • Support for AWS transit gateways on Cisco Cloud APIC.  This would get rid of the limitation of 1.5 Gbps when using VGW
    • Support for using filters to see specific information from AWS flow logs

      •      You can define up to eight filters for a given AWS log group at the same time.

      • We recommend that you configure statistics filters using the Cisco Cloud APIC GUI.

  • Support for statistics collection for AWS transit gateway traffic
    • You must enable collection when you set up Cloud APIC for AWS transit gateway, and you must create flow logs

Let’s summarize the steps needed to add a AWS site to the ACI Fabric:

1. Check / Verify that your AWS account has the required Limits
2. Create Key Pairs for the capic ssh login
3. Register for cAPIC on AWS Market Place
4. Run the AWS Cloud Formation Template and put in the information as required and complete the run
5. Subscribe to vCSRs from AWS Market Place
6. Browse to cAPIC and do the initial configuration
7. Browse to MSO and add in the new AWS Site
8. Download the configuration files for physical Site vCSR routers that you will need to configure for matching IPsec tunnels
9. Configure the on Prem CSRs based on the configs downloaded in previous steps
10. Verify on the CSRs that tunnel has come up, OSPF neighbors are up and bgp l2vpn evpn peers are up
11. Verify from cAPIC that everything is looking good

Let’s now Start on the AWS Side

Step 1: Check / Verify that your AWS account has the required Limits

First Step is to identify the AWS account you will use for the Infra Tenant.  Then login to AWS console on that account with admin privileges.  

Choose a region you will work with

• Go To Services > EC2 > Limits and check for the Elastic IP Limit you have for that account. 
• The Default Limit is 5.  You will need 9 elastic IPs to do this integration. 
• Click on EC2-VPC Elastic IPs if your region supports it, else click on EC2-Classic Elastic IP
• Click on “Request limit Increase”  and open a ticket with AWS to increase the limit.  I would suggest to increast the limit to 12 since you may want to use some later with EC2s if you want permanent Public IPs

Once you open a ticket you should get back an email right away and they will generally get it take care of in 15 to 20 minutes and send you back an email with confirmation

Step 2:  Create Key Pairs for the capic ssh login

Once this is taken care of you can start.  Log into AWS Console, go to Services > EC2 > Key Pairs  and click on Create Key pair

Put in a name for the key pair;  choose pem and Create key pair

Now Download the pem file.    This is your private key file for SSH to cAPIC that you will need to use this key anytime you want to ssh to cAPIC.  So, keep this safely stored in a directory.  ( It is possible to turn off key use for ssh’ing to cAPIC from the cAPIC GUI later, but this is not a good security practice)

Step3:  Register for cAPIC on AWS Market Place

Once this is taken care of you can start.  Log into AWS Console, go to Services and type in market  to search for AWS Marketplace Subscriptions

Click on Discover Products and then type in apic in the search bar.  Click on the first choice

Click Continue to Subscribe

Accept Terms

You will be subscribed now

Click Continue to Configuration

Step 4: Run the AWS Cloud Formation Template and put in the information as required and complete the run

Click Launch

Choose Action to Launch with Cloud Formation Template and Launch

In the next screen, click Next

On Next Screen, put in the Stack details.

• Stack Name
• Fabric Name
• Infra VPC Pool    — choose a unique pool not used in other fabrics

Continue to populate:

• Availability Zone
• Password
• Key Pair
• Access List

Review  Stack

Acknowledge and Create Stack

Watch the Status

Step 5: Subscribe to vCSRs from AWS Market Place

Meanwhile, go back to market place on a second tab (from AWS Console), let this one keep updating status

Search for csr and choose the 1st one

Continue to Subscribe

Accept Terms

When you come to Continue to Configuration, do not Continue.   Exit From here.    DONT CONTINUE.  Otherwise AWS will start spinning up the CSRs.  You want cAPIC to do this..

Go Back to the tab that was updating status

Next, go to EC2 Intances and make sure that Status says running and status checks says 2/2

Also, check for the Pubilc IP

Step 6: Browse to cAPIC and do the initial configuration

Now Log into cAPIC using that Public IP

First Time Setup will pop up

Keep Configuring the items one by one

Choose the Regions where Tenants can Reside.   You can choose the Cloud Routers with Inter-Site Connectivity options which will spin up another pair of Cloud CSRs in the other region, or you can choose the region with no Cloud Rotuers/Inter-Site Connectivity, in which case it will use the AWS VGW connectivity.  The second option is limited to 2 regions unless you open a case with AWS and request for security groups to be increased

New in 5.0(1)k

Transit Gateway is a new feature in release 5.0(1)k.   Using TGW does not limit you to 1.5 Gbps bandwidth as was the case with VGW.  Please ensure to enable TGW and TGW stats if using 5.0(1)k.

Choose the BGP AS number. 

Caution:

• Do not use 65412 as this is used by VGW in AWS / ACI integration
  
• Do not use 64518 as this is used by VNG in Azure / ACI integration

Below is a screenshot of both AWS and Azure integrated ACI Fabric where you can clearly see this.

Going back to our integration...

New in 5.0(1)k

Transit Gateway is a new feature in release 5.0(1)k.   Transit Gateway in cAPIC is called HUB Network.   Please put in a name and bgp AS number for Hub Network

Now, Enter:

• Number of routers per Region
• Username for Cloud CSR
• Password for Cloud CSR
• Throughput for Cloud CSR, all the way to 1Gbps

Enter the License for the Cloud CSR (need to obtain this from https://software.cisco.com)

Now, Specify the IP addresses for the 2 On Premise virtual CSR. Make sure to specify the IPs of the public Address on ASA that has been configured to NAT to the real vCSR

Note:  This part of the configuration has moved to MSO in Release 5.2 cAPIC

Put in the OSPF Area and the External Subnet Prefix Block

Note:  This part of the configuration has moved to MSO in Release 5.2 cAPIC

Click On Summary to Review

Go through the review pages and at end click Finish

Now, Go to the InfraStructure TAB of cAPIC.  You will notice that OSPF, BGP Peers are down and in Error State

Go to the ASA and do a “show conn | i UDP”   You will see that there are IPs trying to establish the IPSec Connectivity on UDP Port 500.  If you look at it properly you will notice that there are 2 IPs and each of the IP from outside is trying to establish IPSec connection to both of the on Prem Virtual CSRs

If you go to AWS console and compare, you will realize that they are the 3rd IP from the public Elastic IP of each of the AWS Cloud CSRs that are trying to do this

If you click on the interface e2 of the CSRs you would see that that  Public IP matches to a Private IP inside the CSR

Ethernet 2 on the AWS console corresponds to G3 on the Cloud CSR router.  Looking at the description on one of the Cloud CSR Rotuers (by first SSH’ing to the Cloud CSR router, you will see that this matches up.  The description of the interface also shows as “cloud external tunnel source”.  So, this interface IP in fact is trying to establish the IPSec Tunnel connection to the On Prem virtual CSR router.  The reason that the IPSec Tunnels are not up yet is because the On Prem virtual CSR routers have not been configured yet for IPSec Connectivity

Step 7: Browse to MSO and add in the new AWS Site

Next Step is to log in to the MSO and add the Cloud Site as a new site

Now, click on Configure Infra and then on the main cloud site and turn on ACI Multi-Site

If you click on each of the Cloud CSR routers ( the equivalent of Spines in physical Fabrics), you will see that configs have been already populated

Let’s go ahead and hit the deploy button to deploy the configs

Step 8: Download the configuration files for physical Site vCSR routers that you will need to configure for matching IPsec tunnels

Now Click on the arrow on Deploy and click on “Download IPN Device Config files only“.   These are actually the corresponding config files that need to go on the on Prem virutal CSRs to complet the IPSec Configurations.  You will notice that there are 4 files.  The reason is that each vCSR on On Prem side will have 2 IPSec tunnels.  One to each Clould CSR on AWS.  Each config file has 1 IPSec tunnel config

The donwloaded files are below

Step 9: Configure the on Prem CSRs based on the configs downloaded in previous steps

Let’s look at our topology again. You will notice that  G2 on the vCSRs face towards the Internet ( for the On Prem vCSRs).

Now Let’s look at the 173.37.213.73, 1st  config file.  This one is the config for vCSR2  IPSec Tunnel 1000.  We need to change the suggested config to match the existing topology of the already configured vCSR

• We repeat the same process and change the other file for 173.37.213.73 which is again for vCSR2 for IPSec Tunnel 1001.   After that we copy both the config to vCSR2.
• Following that we repeat the same process for on Prem vCSR1 ( i.e. modify the suggested configs for the 2 config files and copy them to vCSR1 )

Step 10: Verify on the CSRs that tunnel has come up, OSPF neighbors are up and bgp l2vpn evpn peers are up

Now,  Let’s do some checking from On Prem vCSRs

Step 11: Verify from cAPIC that everything is looking good

Let’s check from the Cloud CSRs

From Spot Checks, things are looking good !

Let’s look from cAPIC to see what the Connectivity Status looks like

As you can see:

• Connectivity Status is O.K
• Deployment Status is : Success
• IPsec Tunnels are up
• OSPF neighbors are up
• BGP Sessions are up

Everything is looking Good !   We are done bringing up the infra Tenant Successfully !   Now we can add tenants as shown in the previous posts I’ve written at this site.

References:

• Cisco Cloud ACI on AWS White Paper
• Cisco Cloud ACI on Microsoft Azure White Paper
• Internet Service for Cisco Cloud APIC Workloads: (how to create untrusted user)
• Cisco Cloud APIC for AWS Installation Guide, Release 4.2(x)
• Shared On-Premises L3Out for Cisco Cloud APIC Workloads
• Cloud APIC Install Guide-Azure
• Cisco Cloud on Azure White Paper
• Cloud APIC Install / Upgrade & Configure Guide