ACI/Cloud Extension Usage Primer (AWS) – Deploying from Scratch- Physical ACI Fabric With ACI Cloud Fabric

In this post, I will show you step by step how to integrate your ACI Physical Fabric with AWS ACI Cloud Fabric.  I would suggest first reading through the introduction section of “ACI/Cloud Extension Usage Primer (AWS) – Cloud Tenant Only (deploying Application Load Balancer with Service Graph on AWS)“.   That will give you a very good understanding of the components used for ACI Cloud Integration.

Before we start, let’s quickly discuss the main idea of how to set up the connectivity.    Let’s take a quick look at what you would do for connectivity if you had 2 ACI Physical Fabrics that you would put together to create a ACI Multisite fabric.    This is pretty straight forward.  To make the diagram less messy and to convey the idea,  I only have 1 APIC, 1 Spine and 2 leaves in each fabric.  For a production size fabric with multiple spines / APIC clusters and MSO clusters the idea is the same.

Figure 1

All of you can relate to the above easily and most of you reading this have probably already implemented this.  Most people need a DCI solution and ACI Multisite gives that to you very easily.   The main points to note here are that:

  • Spines of each site connect to their  ISNs and run OSPF between Spines and ISN
  • ISNs connect to the IPv4 cloud and you need to make sure that spine to spine routing connectivity is good
  • MSO cluster should be able to reach both Site’s APICs OOB
  • Once you setup Multisite through MSO, you will have BGP eVPN peering between spines of different sites.

When Setting up one or more Physical ACI Fabric to integrate with Cloud ACI Fabric, the idea is exactly the same.  However since you are now connecting over the public Ipv4 Network, you have to give the physical connectivity a little bit more thought to logically achieve the same sort of topology.   Let’s take a look at the figure below

Figure 2

You can easily see that the connectivity is very similar.   Some things to note here are:

  • On the AWS side, you don’t have real Spines and Leaves and APICs.   You have a couple of Cloud virtual CSRs and a Cloud APIC also known as cAPIC.  You could think of the AWS side CSRs as taking the place of the Spines on the cloud side.  All of those components will need to have Internet connectivity and this will be done by using AWS Elastic IPs.  (Elastic IPs in AWS are nailed down public IPs that will not change when you reboot your ec2 instances (or any other instances that you run on AWS)
  • We are connecting the physical fabric to the Cloud fabric over the public Internet (generally unless using AWS direct connect). So, it’s common sense that we will want to front end the fabric with a Firewall (cluster).
  • The firewall needs to be configured to do NAT/PAT translations so that outside public addresses gets translated to inside addresses.   You specifically need to do NAT tranalation, IP to IP for APIC, MSO and anything else you may want to access directly from the outside like jump boxes, etc, etc.
  • Behind the firewall cluster you would have a couple of vCSRs running on a couple of hypervisors (for redundancy of course). 
  • What happens is that we bring up a full mesh of IPSec tunnels between the vCSRs on the physical side and the Cloud vCSRs on the AWS side.  
  • The IPSec tunnels act as point to point links between the AWS Cloud CSRs and the physical Fabric side Cloud virtual CSRs.  
  • Now your OSPF peering is able to form neighbor relationships all the way from cloud CSRs to physical Fabric vCSRs through the IPSec tunnels.
  • Now for control plane BGP eVPN peerings are built from the physical spines to the cloud CSRs on AWS side.
  • for data plane the vxlan tunnels are built from physical spine to cloud CSRs

I’m hoping that you notice that the physical connectivity is very similar with some minor additions.   Now,  if you think about this a little bit more, you have to do your cabling in such a way that the virtual CSRs on the physical side is in front of the ISN routers and behind the Firewalls.

Below is a simple way of achieving this.


Figure 3

The diagram is pretty self explanatory.   Some items I want to point out are:

  • we make 2 VRFs in the ISN routers 
  • if you study the diagram you will notice that vmware DVS port group DMZ-ISN has all the virtual components (like vCSRs, MSO, jumpboxes, etc, etc) that need to reach out to the internet / AWS fabric components
  • The Spines connect to the ISN in the VRF Spine defined on the ISN.
  • However for the Spine to reach out to the Internet, it first has to get routed through the virtual CSRs, because of the way the VRFs segregate the ISNs.   Recall that the virtual CSRs will have ipSEC tunnels going to the Cloud CSRs.
  • MSO can reach both APICs (physical APIC and cAPIC in AWS (through the internet).  Note, that you could also place the MSO on the AWS cloud itself running on the virtual Service Engine (found through AWS console in AWS market place).  If you place the MSO on the physical Fabric side, like shown in this diagram you also have the option of running the MSO on the virtual Service Engine )based on Kubernetes Cluster).   If interested please read Deploying MSO on Cisco Application Service Engine (OVA based SE).

Once you are done cabling up like shown in the above diagram, your logical diagram on the on premise (physical side) will look like the below.

Figure 4

In the above logical diagram, the 2 ISNs are the same ISN but different VRFs in that ISN.   Separating out this way  makes it’s easy to visualize the connectivity. 

On the vCSR’s please make sure:

  • to point the default gateway to the SVI of VRF ISN-DMZ,  ( in my example.  
  • On the VRF -DMZ please make sure to point the default gateway to the Connected ASA interface in my case. 
  • You could try to ping some destination on the Internet from the vCSR like to make sure it’s working (given that your ASA is configured correctly for NAT and icmp inspection).

When you install the vCSRs on the on Prem Fabric side, please make sure that you license for the security feature of virtual CSR.   That has the IPSec capability and you will obviously need to bring up ipSec Tunnels from your virtual CSR.   Please See the Licensing CSR1000V guide on CCO.

Below is a excerpt from that licensing document.

Figure 5

Note:   During the install of the vCSR OVA you will be asked to choose the feature set you want.  If you happen to not have chosen “Security” or “AX” based feature set, you could install the security feature set after booting up the vCSR by executing the command “license boot level security” from the config prompt.   You will then need to reboot the vCSR for it to take effect.

Figure 6

The next step will be to configure the Spine interface facing the ISN and bring up OSPF on that interface.   You will have to configure the following:

First make sure that you configure your Fabric Access Policies and tie the interface on the spine (going to ISN) to a vlan pool containing vlan 4.

If the physical fabric is already part of a Multisite fabric, this has already been done.

Figure 7

Next using MSO add in the Physical Site

If the physical fabric is already part of a Multisite fabric, this has already been done.

Figure 8

Next Fill in the details in the form

If the physical fabric is already part of a Multisite fabric, this has already been done.

Figure 9

Next Click on Configure Infra

If the physical fabric is already part of a Multisite fabric, this has already been done.

Figure 10

Next, click on the Site just Added

If the physical fabric is already part of a Multisite fabric, this has already been done.

Figure 11

Next, fill in the information as requested:

If the physical fabric is already part of a Multisite fabric, this has already been done.

  1. Click on the main Site Name
  2. Turn on MultiSite
  3. Put in Site ID (1 in my case)
  4. Put in Overlay Multicast TEP ( this is the Head End Replication Unicast IP)
  5. Put in BGP AS
  6. Put in OSPF Info
Figure 12

Next Click on POD # and fill in required info

If the physical fabric is already part of a Multisite fabric, this has already been done.

  1. click on POD #
  2. Put in the Overlay TEP  in my case.  This is the Unicast Any Cast TEP IP.  BGP EVPN next hop will be set to this for the POD
Figure 13

Next, Click on Spines (one at a time) and fill in Required Information

If the physical fabric is already part of a Multisite fabric, this has already been done.

  1. Click on Spines (one at a time)
  2. Put in Interface #, IP and MTU
  3. Turn On BGP Peering
  4. BGP EVPN Router ID.  This will be a loopback on the Spine and is also known as CP TEP (control Plane TEP).  BGP will use this loopback to source the peering
  5. Make Spine Route Reflector
  6. Click Deploy
Figure 14

Next, go to the Physical APIC, Navigate to Infra Tenant, Fabric ext Connection Policies:

  1. Put in the Extended Community, in my case extended:as2-nn4:5:16
  2. Verify that POD ID  and Unicast Data Plane TEP is showing up there
Figure 15

Now, you are done configuring the spine Interface side.  Next, configure the ISN Interface connectint to spine and put it in matching OSPF config on that side.

Figure 16
Figure 17

On the ISN Router, check to make sure that the OSPF Neighbor to Spine has come up

Figure 18

Now, extend OSPF all the way to the vCSR Interfaces

Figure 19

Configure and check for neighbors on both vCSR Routers

Figure 20

One more Check you may want to do:

Please verify that the ASA firewall is allowing the following:

  • tcp port 22 for ssh
  • tcp port 443 for ssl
  • udp port 500 and udp port 4500 for IPSec tunnels

You can use different methods to do this verification.  What I like to do for fast check,  is:

  • for tcp 22,  use netcat or just ssh into the ubuntu box from outside
  • for tcp, 443, use netcat or just bring up a docker container with nginx and browse to it

O.K. now, all the Physical Side Prep work is done !

Let’s summarize the steps needed to add a AWS site to the ACI Fabric:

  1. Check / Verify that your AWS account has the required Limits
  2. Create Key Pairs for the capic ssh login
  3. Register for cAPIC on AWS Market Place
  4. Run the AWS Cloud Formation Template and put in the information as required and complete the run
  5. Subscribe to vCSRs from AWS Market Place
  6. Browse to cAPIC and do the initial configuration
  7. Browse to MSO and add in the new AWS Site
  8. Download the configuration files for physical Site vCSR routers that you will need to configure for matching IPsec tunnels
  9. Configure the on Prem CSRs based on the configs downloaded in previous steps
  10. Verify on the CSRs that tunnel has come up, OSPF neighbors are up and bgp l2vpn evpn peers are up
  11. Verify from cAPIC that everything is looking good

Let’s now Start on the AWS Side

First Step is to identify the AWS account you will use for the Infra Tenant.  Then login to AWS console on that account with admin privileges.  

Choose a region you will work with

  • Go To Services > EC2 > Limits and check for the Elastic IP Limit you have for that account. 
  • The Default Limit is 5.  You will need 9 elastic IPs to do this integration. 
  • Click on EC2-VPC Elastic IPs if your region supports it, else click on EC2-Classic Elastic IP
  • Click on “Request limit Increase”  and open a ticket with AWS to increase the limit.  I would suggest to increast the limit to 12 since you may want to use some later with EC2s if you want permanent Public IPs

Once you open a ticket you should get back an email right away and they will generally get it take care of in 15 to 20 minutes and send you back an email with confirmation

Figure 21

Once this is taken care of you can start.  Log into AWS Console, go to Services > EC2 > Key Pairs  and click on Create Key pair

Figure 21a

Put in a name for the key pair;  choose pem and Create key pair

Figure 21b

Now Download the pem file.    This is your private key file for SSH to cAPIC that you will need to use this key anytime you want to ssh to cAPIC.  So, keep this safely stored in a directory.  ( It is possible to turn off key use for ssh’ing to cAPIC from the cAPIC GUI later, but this is not a good security practice)

Figure 21c

Once this is taken care of you can start.  Log into AWS Console, go to Services and type in market  to search for AWS Marketplace Subscriptions

Figure 22

Click on Discover Products and then type in apic in the search bar.  Click on the first choice

Figure 23

Click Continue to Subscribe

Figure 24

Accept Terms

Figure 25

You will be subscribed now

Figure 26

Click Continue to Configuration

Figure 27

Click Lanuch

Figure 28

Choose Action to Launch with Cloud Formation Template and Launch

Figure 29

In the next screen, click Next

Figure 30

On Next Screen, put in the Stack details.

  • Stack Name
  • Fabric Name
  • Infra VPC Pool    — choose a unique pool not used in other fabrics
Figure 31

Continue to populate:

  • Availability Zone
  • Password
  • Key Pair
  • Access List
Figure 32

Review  Stack

Figure 33
Figure 33a
Figure 33b

Acknowledge and Create Stack

Figure 33c

Watch the Status

Figure 34

Meanwhile, go back to market place on a second tab (from AWS Console), let this one keep updating status

Figure 35

Search for csr and choose the 1st one

Figure 36

Continue to Subscribe

Figure 37

Accept Terms

Figure 38

When you come to Continue to Configuration, do not Continue.   Exit From here.    DONT CONTINUE.  Otherwise AWS will start spinning up the CSRs.  You want cAPIC to do this..

Figure 39

Go Back to the tab that was updating status

Figure 40

Next, go to EC2 Intances and make sure that Status says running and status checks says 2/2

Also, check for the Pubilc IP

Figure 41

Now Log into cAPIC using that Public IP

Figure 42

First Time Setup will pop up

Figure 43

Keep Configuring the items one by one

Figure 44
Figure 44a
Figure 44b

Choose the Regions where Tenants can Reside.   You can choose the Cloud Routers with Inter-Site Connectivity options which will spin up another pair of Cloud CSRs in the other region, or you can choose the region with no Cloud Rotuers/Inter-Site Connectivity, in which case it will use the AWS VGW connectivity.  The second option is limited to 2 regions unless you open a case with AWS and request for security groups to be increased

Figure 44c
Figure 44d

Choose the BGP AS number. 


  • Do not use 65412 as this is used by VGW in AWS / ACI integration
  • Do not use 64518 as this is used by VNG in Azure / ACI integration

Below is a screenshot of both AWS and Azure integrated ACI Fabric where you can clearly see this.

Figure 44e

Going back to our integration...

Figure 45

Now, Enter:

  • Number of routers per Region
  • Username for Cloud CSR
  • Password for Cloud CSR
  • Throughput for Cloud CSR, all the way to 1Gbps
Figure 46

Enter the License for the Cloud CSR (need to obtain this from

Figure 47

Now, Specify the IP addresses for the 2 On Premise virtual CSR. Make sure to specify the IPs of the public Address on ASA that has been configured to NAT to the real vCSR

Figure 48

Put in the OSPF Area and the External Subnet Prefix Block

Figure 49

Click On Summary to Review

Figure 50

Go through the review pages and at end click Finish

Figure 51

Now, Go to the InfraStructure TAB of cAPIC.  You will notice that OSPF, BGP Peers are down and in Error State

Figure 51
Figure 52

Go to the ASA and do a “show conn | i UDP”   You will see that there are IPs trying to establish the IPSec Connectivity on UDP Port 500.  If you look at it properly you will notice that there are 2 IPs and each of the IP from outside is trying to establish IPSec connection to both of the on Prem Virtual CSRs

Figure 53

If you go to AWS console and compare, you will realize that they are the 3rd IP from the public Elastic IP of each of the AWS Cloud CSRs that are trying to do this

Figure 54
Figure 55

If you click on the interface e2 of the CSRs you would see that that  Public IP matches to a Private IP inside the CSR

Figure 56
Figure 56b

Ethernet 2 on the AWS console corresponds to G3 on the Cloud CSR router.  Looking at the description on one of the Cloud CSR Rotuers (by first SSH’ing to the Cloud CSR router, you will see that this matches up.  The description of the interface also shows as “cloud external tunnel source”.  So, this interface IP in fact is trying to establish the IPSec Tunnel connection to the On Prem virtual CSR router.  The reason that the IPSec Tunnels are not up yet is because the On Prem virtual CSR routers have not been configured yet for IPSec Connectivity

Figure 57
Figure 58
Figure 59

Next Step is to log in to the MSO and add the Cloud Site as a new site

Figure 60
Figure 61

Now, click on Configure Infra and then on the main cloud site, put in the

Figure 62

If you click on each of the Cloud CSR routers ( the equivalent of Spines in physical Fabrics), you will see that configs have been already populated

Figure 63
Figure 64

Let’s go ahead and hit the deploy button to deploy the configs

Now Click on the arrow on Deploy and click on “Download IPN Device Config files only“.   These are actually the corresponding config files that need to go on the on Prem virutal CSRs to complet the IPSec Configurations.  You will notice that there are 4 files.  The reason is that each vCSR on On Prem side will have 2 IPSec tunnels.  One to each Clould CSR on AWS.  Each config file has 1 IPSec tunnel config

Figure 65
Figure 66

The donwloaded files are below

Figure 67

Let’s look at our topology again. You will notice that  G2 on the vCSRs face towards the Internet ( for the On Prem vCSRs).

Figure 68

Now Let’s look at the, 1st  config file.  This one is the config for vCSR2  IPSec Tunnel 1000.  We need to change the suggested config to match the existing topology of the already configured vCSR

figure 69
  • We repeat the same process and change the other file for which is again for vCSR2 for IPSec Tunnel 1001.   After that we copy both the config to vCSR2.
  • Following that we repeat the same process for on Prem vCSR1 ( i.e. modify the suggested configs for the 2 config files and copy them to vCSR1 )

Now,  Let’s do some checking from On Prem vCSRs

Figure 70
Figure 71
Figure 72

Let’s check from the Cloud CSRs

Figure 73
Figure 74
Figure 75

From Spot Checks, things are looking good !

Let’s look from cAPIC to see what the Connectivity Status looks like

Figure 76

As you can see:

  • Connectivity Status is O.K
  • Deployment Status is : Success
  • IPsec Tunnels are up
  • OSPF neighbors are up
  • BGP Sessions are up

Everything is looking Good !   We are done bringing up the infra Tenant Successfully !   Now we can add tenants as shown in the previous posts I’ve written at this site.


Posted in All

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.