Simple Troubleshooting Steps for Cloud ACI/AWS To OnPrem ACI For End Point Reachability Issues

Table of Contents:

  1. Introduction
  2. 2 Items to check, Control Plane & Data Plane
    2.a.Verifying Control Plane
    2.b. Verifying Data Plane
    2.c. Verify evpn on spine
  3. References


In this writeup I will go though some very simple troubleshooting steps that you can follow if you are having issues between ACI/AWS fabric and onPrem Fabric Tenant endpoint reachability.

Regardless of whether you are using Internet Connectivity for the connection or DX Connectivity these troubleshooing steps should help. The assumption is that you have already brought up cAPIC Fabric on AWS, done the necessary configurations from NDO, brought up Tenants and objects and you cannot reach from AWS endpoint (EC2) to onPrem endpoint (let’s say a VM).

2 Items to check, Control Plane & Data Plane

The important thing to note for troubleshooting, is that you have to look at 2 aspects:

  • Control Plane
  • Data Plane

The Control Plane is BGP EVPN between ACI OnPrem Spine and C8Kv routers.

The Data Plane is VXLAN

If any of them are broken you will have reachability problems.

Verifying Control Plane

a) check onprem leaf route table
Verify that you can see cloud cidr with next-hop is CSR Gig4 interface
show ip route vrf userVRF
b) check cloud vpc egress route tables (on AWS Tenant Account)
onprem subnet -> tgw
c) check tgw route table -> if they learn the route 0/0 through CSR (on AWS Infra Account)
d) check NSG rule
e) check zoning rule on onPrem

Below are screenshots from my lab on the items mentioned above.

a) Verify that the Tenant VRF shows the Cloud CIDR being advertised with NH of Gig4 of C8KV

C8KV has ip of
Figure 1: Looking at G4 IP of C8KV

Verifying that my Tenant VRF shows the VPC CIDR of with NH of
Figure 2: Verifying on ACI Leaf that the next hop shows VPC CIDR with correct next hop

b) On Tenant AWS Account, Check the Route Table of the EC2 Subnet to verify that the onPrem subnet shows up with NH of tgw

Figure 3: Route Table of EC2 shows onPrem prefix with next hop of TGW

c) Checking TGW Route Table on Infra AWS Account to verify 0/0 is learnt through Connect attachment
Figure 4: Checking TGW route Table on Infra account for 0/0

Also Verify that the Connect Attachment connects to CSR G2

Figure 5: TGW Connect Attachment peers with C8KV Gig 2

d) check NSG rules
Make sure your ingress and egress rules are configued correctly for the EC2, (based on your contract configurations from NDO)

Figure 6: Checking NSG rules

e) check zoning rule on onPrem
Please see the below writeup to see how to verify the zoning rules (Normally this should not be a problem unless it’s misconfigured)

Verifying Data Plane

If the Control Plane looks good, but your reachability is still not working, chances are that the problem is in the Data Plane.

📙 It’s important to remember the following:

From: To: VXLAN Tunnel
OnPrem Cloud onPrem Leaf –> C8Kv
Cloud onPrem c8Kv –> onPrem Spine

Below is the quick list of items to verify:

a. try to ping CSR g4 from onprem leaf
b. from CSR: ping BGP EVPN RID of spine
c. from CSR: show ip route
-> you should see onprem Infra Pool
d. Show nve peers on the Cat8kv. All these should be up. (these are the vxlan tunnels)
e. ping from onPrem and capture packets on the C8KV to verify if packets are coming in by using packet trace

Below are screenshots from my lab on the items mentioned above.

a) try to ping CSR g4 from onprem leaf

Looking for G4 IP on C8KV
Figure 7: Looking at Gig4 IP on C8KV

Pinging G4 IP from Leaf:

Figure 8: Pinging G4 IP from Leaf

b) from CSR: ping BGP EVPN RID of spine

First Get BGP RID of spine

Figure 9: Obtaining BGP RID of Spine

Pinging the RID of Spine from C8KV
Figure10: Pinging the BGP RID of Spine from C8KV

c) Checking for TEP Pool in Routing table of C8KV

Quick way to look at TEP Pool:
on onPrem APIC do:

cat /data/data_admin/sam_exported.config

Figure 11: Determining the TEP Pool configured on onPrem APIC

You can also check that the tep pools are used for the Destination TEPs from the leaf

show isis dtep vrf overlay-1

Figure 12: show isis dtep vrf overlay-1

Now check that those TEP routes shows from cloud C8KV

show ip route | i 10.7   # (notice its coming from Tu6 in this case)

Figure 13: TEP Pool Prefixes show on routing table of C8KV

d) show NVE Peers (to veify that VXLAN tunnels have established)
Figure 14: looking at NVE Peers

e) ping from onPrem and capture packets on the C8KV to verify if packets are coming in

I show using this at:
Please see right after Figure 41 in the writeup link above..

The commands on the CSRs to be used are as follows:

    • debug platform packet-trace packet 128
    • debug platform condition ipv4 both
    • debug platform condition start
    • debug platform packet-trace packet 128
 To View:
    • show platform packet-trace statistics
    • show platform packet-trace summary
 To Stop:
    • debug platform condition stop
 Other Useful Commands:
    • show platform packet-trace code                   # Show packet-trace drop, inject or punt codes
    • show platform packet-trace configuration     #Show packet-trace debug configuration
    • show platform packet-trace packet     #Per packet details for traced packets
    • show platform packet-trace statistics     #Statistics for packets traced and packet disposition
    • show platform packet-trace summary     #Per packet summary information for traced packets
    • clear platform packet-trace configuration
    • clear platform packet-trace statistics

Verify evpn on spine

On the onPrem Spine use the commands listed to verify evpn network.

show bgp internal limited-vrf
show bgp internal network vrf name_of_vrf

Figure 15: Verifying evpn on Spine


Cloud ACI Documentation

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.