Site icon

ACI/Cloud Extension Usage Primer (AWS) – Deploying from Scratch- Physical ACI Fabric With ACI Cloud Fabric

Soumitra Mukherji

In this post, I will show you step by step how to integrate your ACI Physical Fabric with AWS ACI Cloud Fabric.  I would suggest first reading through the introduction section of “ACI/Cloud Extension Usage Primer (AWS) – Cloud Tenant Only (deploying Application Load Balancer with Service Graph on AWS)“.   That will give you a very good understanding of the components used for ACI Cloud Integration.

Before we start, let’s quickly discuss the main idea of how to set up the connectivity.    Let’s take a quick look at what you would do for connectivity if you had 2 ACI Physical Fabrics that you would put together to create a ACI Multisite fabric.    This is pretty straight forward.  To make the diagram less messy and to convey the idea,  I only have 1 APIC, 1 Spine and 2 leaves in each fabric.  For a production size fabric with multiple spines / APIC clusters and MSO clusters the idea is the same.

Figure 1

All of you can relate to the above easily and most of you reading this have probably already implemented this.  Most people need a DCI solution and ACI Multisite gives that to you very easily.   The main points to note here are that:

When Setting up one or more Physical ACI Fabric to integrate with Cloud ACI Fabric, the idea is exactly the same.  However since you are now connecting over the public Ipv4 Network, you have to give the physical connectivity a little bit more thought to logically achieve the same sort of topology.   Let’s take a look at the figure below

Figure 2

You can easily see that the connectivity is very similar.   Some things to note here are:

I’m hoping that you notice that the physical connectivity is very similar with some minor additions.   Now,  if you think about this a little bit more, you have to do your cabling in such a way that the virtual CSRs on the physical side is in front of the ISN routers and behind the Firewalls.

Below is a simple way of achieving this.

 

Figure 3

The diagram is pretty self explanatory.   Some items I want to point out are:

Once you are done cabling up like shown in the above diagram, your logical diagram on the on premise (physical side) will look like the below.

Figure 4

In the above logical diagram, the 2 ISNs are the same ISN but different VRFs in that ISN.   Separating out this way  makes it’s easy to visualize the connectivity. 

On the vCSR’s please make sure:

When you install the vCSRs on the on Prem Fabric side, please make sure that you license for the security feature of virtual CSR.   That has the IPSec capability and you will obviously need to bring up ipSec Tunnels from your virtual CSR.   Please See the Licensing CSR1000V guide on CCO.

Below is a excerpt from that licensing document.

Figure 5

Note:   During the install of the vCSR OVA you will be asked to choose the feature set you want.  If you happen to not have chosen “Security” or “AX” based feature set, you could install the security feature set after booting up the vCSR by executing the command “license boot level security” from the config prompt.   You will then need to reboot the vCSR for it to take effect.

Figure 6

The next step will be to configure the Spine interface facing the ISN and bring up OSPF on that interface.   You will have to configure the following:

First make sure that you configure your Fabric Access Policies and tie the interface on the spine (going to ISN) to a vlan pool containing vlan 4.

If the physical fabric is already part of a Multisite fabric, this has already been done.

Figure 7

Next using MSO add in the Physical Site

If the physical fabric is already part of a Multisite fabric, this has already been done.

Figure 8

Next Fill in the details in the form

If the physical fabric is already part of a Multisite fabric, this has already been done.

Figure 9

Next Click on Configure Infra

If the physical fabric is already part of a Multisite fabric, this has already been done.

Figure 10

Next, click on the Site just Added

If the physical fabric is already part of a Multisite fabric, this has already been done.

Figure 11

Next, fill in the information as requested:

If the physical fabric is already part of a Multisite fabric, this has already been done.

  1. Click on the main Site Name
  2. Turn on MultiSite
  3. Put in Site ID (1 in my case)
  4. Put in Overlay Multicast TEP ( this is the Head End Replication Unicast IP)
  5. Put in BGP AS
  6. Put in OSPF Info
Figure 12

Next Click on POD # and fill in required info

If the physical fabric is already part of a Multisite fabric, this has already been done.

  1. click on POD #
  2. Put in the Overlay TEP  100.100.100.120  in my case.  This is the Unicast Any Cast TEP IP.  BGP EVPN next hop will be set to this for the POD
Figure 13

Next, Click on Spines (one at a time) and fill in Required Information

If the physical fabric is already part of a Multisite fabric, this has already been done.

  1. Click on Spines (one at a time)
  2. Put in Interface #, IP and MTU
  3. Turn On BGP Peering
  4. BGP EVPN Router ID.  This will be a loopback on the Spine and is also known as CP TEP (control Plane TEP).  BGP will use this loopback to source the peering
  5. Make Spine Route Reflector
  6. Click Deploy
Figure 14

Next, go to the Physical APIC, Navigate to Infra Tenant, Fabric ext Connection Policies:

  1. Put in the Extended Community, in my case extended:as2-nn4:5:16
  2. Verify that POD ID  and Unicast Data Plane TEP is showing up there
Figure 15

Now, you are done configuring the spine Interface side.  Next, configure the ISN Interface connectint to spine and put it in matching OSPF config on that side.

Figure 16
Figure 17

On the ISN Router, check to make sure that the OSPF Neighbor to Spine has come up

Figure 18

Now, extend OSPF all the way to the vCSR Interfaces

Figure 19

Configure and check for neighbors on both vCSR Routers

Figure 20

One more Check you may want to do:

Please verify that the ASA firewall is allowing the following:

You can use different methods to do this verification.  What I like to do for fast check,  is:

Below I show you some steps for quick verification on ASA.

Example: Checking for UDP 1812 to see if it comes all the way to ASA Permeter Firewall through corporate Firewall

Figure 20a

Example: Checking for tcp 1812 to see if it comes all the way to ASA Permeter Firewall through corporate Firewall (steps 1 and 2 are the same as above, so if you did the above, you can go directly to step 3)

Figure 20b

O.K. now, all the Physical Side Prep work is done !

Updates:

Let’s summarize the steps needed to add a AWS site to the ACI Fabric:

  1. Check / Verify that your AWS account has the required Limits
  2. Create Key Pairs for the capic ssh login
  3. Register for cAPIC on AWS Market Place
  4. Run the AWS Cloud Formation Template and put in the information as required and complete the run
  5. Subscribe to vCSRs from AWS Market Place
  6. Browse to cAPIC and do the initial configuration
  7. Browse to MSO and add in the new AWS Site
  8. Download the configuration files for physical Site vCSR routers that you will need to configure for matching IPsec tunnels
  9. Configure the on Prem CSRs based on the configs downloaded in previous steps
  10. Verify on the CSRs that tunnel has come up, OSPF neighbors are up and bgp l2vpn evpn peers are up
  11. Verify from cAPIC that everything is looking good

Let’s now Start on the AWS Side

Step 1: Check / Verify that your AWS account has the required Limits

First Step is to identify the AWS account you will use for the Infra Tenant.  Then login to AWS console on that account with admin privileges.  

Choose a region you will work with

Once you open a ticket you should get back an email right away and they will generally get it take care of in 15 to 20 minutes and send you back an email with confirmation

Figure 21

Step 2:  Create Key Pairs for the capic ssh login

Once this is taken care of you can start.  Log into AWS Console, go to Services > EC2 > Key Pairs  and click on Create Key pair

Figure 21a

Put in a name for the key pair;  choose pem and Create key pair

Figure 21b

Now Download the pem file.    This is your private key file for SSH to cAPIC that you will need to use this key anytime you want to ssh to cAPIC.  So, keep this safely stored in a directory.  ( It is possible to turn off key use for ssh’ing to cAPIC from the cAPIC GUI later, but this is not a good security practice)

Figure 21c

Step3:  Register for cAPIC on AWS Market Place

Once this is taken care of you can start.  Log into AWS Console, go to Services and type in market  to search for AWS Marketplace Subscriptions

Figure 22

Click on Discover Products and then type in apic in the search bar.  Click on the first choice

Figure 23

Click Continue to Subscribe

Figure 24

Accept Terms

Figure 25

You will be subscribed now

Figure 26

Click Continue to Configuration

Figure 27

Step 4: Run the AWS Cloud Formation Template and put in the information as required and complete the run

Click Launch

Figure 28

Choose Action to Launch with Cloud Formation Template and Launch

Figure 29

In the next screen, click Next

Figure 30

On Next Screen, put in the Stack details.

Figure 31

Continue to populate:

Figure 32

Review  Stack

Figure 33
Figure 33a
Figure 33b

Acknowledge and Create Stack

Figure 33c

Watch the Status

Figure 34

Step 5: Subscribe to vCSRs from AWS Market Place

Meanwhile, go back to market place on a second tab (from AWS Console), let this one keep updating status

Figure 35

Search for csr and choose the 1st one

Figure 36

Continue to Subscribe

Figure 37

Accept Terms

Figure 38

When you come to Continue to Configuration, do not Continue.   Exit From here.    DONT CONTINUE.  Otherwise AWS will start spinning up the CSRs.  You want cAPIC to do this..

Figure 39

Go Back to the tab that was updating status

Figure 40

Next, go to EC2 Intances and make sure that Status says running and status checks says 2/2

Also, check for the Pubilc IP

Figure 41

Step 6: Browse to cAPIC and do the initial configuration

Now Log into cAPIC using that Public IP

Figure 42

First Time Setup will pop up

Figure 43

Keep Configuring the items one by one

Figure 44
Figure 44a
Figure 44b

Choose the Regions where Tenants can Reside.   You can choose the Cloud Routers with Inter-Site Connectivity options which will spin up another pair of Cloud CSRs in the other region, or you can choose the region with no Cloud Rotuers/Inter-Site Connectivity, in which case it will use the AWS VGW connectivity.  The second option is limited to 2 regions unless you open a case with AWS and request for security groups to be increased

Figure 44c
Figure 44d

New in 5.0(1)k

Transit Gateway is a new feature in release 5.0(1)k.   Using TGW does not limit you to 1.5 Gbps bandwidth as was the case with VGW.  Please ensure to enable TGW and TGW stats if using 5.0(1)k.

Figure 44da

Choose the BGP AS number. 

Caution:

  • Do not use 65412 as this is used by VGW in AWS / ACI integration
  • Do not use 64518 as this is used by VNG in Azure / ACI integration

Below is a screenshot of both AWS and Azure integrated ACI Fabric where you can clearly see this.

Figure 44e

Going back to our integration...

Figure 45

New in 5.0(1)k

Transit Gateway is a new feature in release 5.0(1)k.   Transit Gateway in cAPIC is called HUB Network.   Please put in a name and bgp AS number for Hub Network

Figure 45a

Now, Enter:

Figure 46

Enter the License for the Cloud CSR (need to obtain this from https://software.cisco.com)

Figure 47

Now, Specify the IP addresses for the 2 On Premise virtual CSR. Make sure to specify the IPs of the public Address on ASA that has been configured to NAT to the real vCSR

Note:  This part of the configuration has moved to MSO in Release 5.2 cAPIC

Figure 48

Put in the OSPF Area and the External Subnet Prefix Block

Note:  This part of the configuration has moved to MSO in Release 5.2 cAPIC

Figure 49

Click On Summary to Review

Figure 50

Go through the review pages and at end click Finish

Figure 51

Now, Go to the InfraStructure TAB of cAPIC.  You will notice that OSPF, BGP Peers are down and in Error State

Figure 51
Figure 52

Go to the ASA and do a “show conn | i UDP”   You will see that there are IPs trying to establish the IPSec Connectivity on UDP Port 500.  If you look at it properly you will notice that there are 2 IPs and each of the IP from outside is trying to establish IPSec connection to both of the on Prem Virtual CSRs

Figure 53

If you go to AWS console and compare, you will realize that they are the 3rd IP from the public Elastic IP of each of the AWS Cloud CSRs that are trying to do this

Figure 54
Figure 55

If you click on the interface e2 of the CSRs you would see that that  Public IP matches to a Private IP inside the CSR

Figure 56
Figure 56b

Ethernet 2 on the AWS console corresponds to G3 on the Cloud CSR router.  Looking at the description on one of the Cloud CSR Rotuers (by first SSH’ing to the Cloud CSR router, you will see that this matches up.  The description of the interface also shows as “cloud external tunnel source”.  So, this interface IP in fact is trying to establish the IPSec Tunnel connection to the On Prem virtual CSR router.  The reason that the IPSec Tunnels are not up yet is because the On Prem virtual CSR routers have not been configured yet for IPSec Connectivity

Figure 57
Figure 58
Figure 59

Step 7: Browse to MSO and add in the new AWS Site

Next Step is to log in to the MSO and add the Cloud Site as a new site

Figure 60
Figure 61

Now, click on Configure Infra and then on the main cloud site and turn on ACI Multi-Site

Figure 62

If you click on each of the Cloud CSR routers ( the equivalent of Spines in physical Fabrics), you will see that configs have been already populated

Figure 63
Figure 64

Let’s go ahead and hit the deploy button to deploy the configs

Step 8: Download the configuration files for physical Site vCSR routers that you will need to configure for matching IPsec tunnels

Now Click on the arrow on Deploy and click on “Download IPN Device Config files only“.   These are actually the corresponding config files that need to go on the on Prem virutal CSRs to complet the IPSec Configurations.  You will notice that there are 4 files.  The reason is that each vCSR on On Prem side will have 2 IPSec tunnels.  One to each Clould CSR on AWS.  Each config file has 1 IPSec tunnel config

Figure 65
Figure 66

The donwloaded files are below

Figure 67

Step 9: Configure the on Prem CSRs based on the configs downloaded in previous steps

Let’s look at our topology again. You will notice that  G2 on the vCSRs face towards the Internet ( for the On Prem vCSRs).

Figure 68

Now Let’s look at the 173.37.213.73, 1st  config file.  This one is the config for vCSR2  IPSec Tunnel 1000.  We need to change the suggested config to match the existing topology of the already configured vCSR

figure 69

Step 10: Verify on the CSRs that tunnel has come up, OSPF neighbors are up and bgp l2vpn evpn peers are up

Now,  Let’s do some checking from On Prem vCSRs

Figure 70
Figure 71
Figure 72

Step 11: Verify from cAPIC that everything is looking good

Let’s check from the Cloud CSRs

Figure 73
Figure 74
Figure 75

From Spot Checks, things are looking good !

Let’s look from cAPIC to see what the Connectivity Status looks like

Figure 76

As you can see:

Everything is looking Good !   We are done bringing up the infra Tenant Successfully !   Now we can add tenants as shown in the previous posts I’ve written at this site.

References:

Exit mobile version