Table of Contents:
- Introduction
- Adding 9364C-GX switch to existing ACI Fabric
- Adding N9364C-GX switch as a Spine during brand new/initial ACI Fabric deployment
- References
Introduction
Cisco Nexus 9364C-GX can be used as either a ACI Spine or ACI leaf, giving you great flexibility. However, you should be aware of how to complete Fabric Discovery / joining the Node to the ACI fabric.
The diagram below shows the front panel and ports available on the N9364C-GX
Figure 1: N9364-C Front Panel Ports
Adding N9364C-GX switch to existing ACI Fabric
- The default mode of the N9364C-GX Switch with ACI Code is to function as a leaf.
- The uplink ports (fabric connectivity) on that leaf switch are the last 2 ports (63 and 64).
- Ports 1 through 62 are downlink ports (host connectivity, APIC also could connect there) on that leaf switch
📙 The N9364C-GX has 2 built in SFP ports where you can connect your APICs to. Each APIC has 2 connections (active/standby). In case you wanted to connect your cluster of 3 APICs to 2 Leaf Role N9364C-GX, you would need an extra SFP Port in each of the leaves. For this you can use QSA-QSFP to SFP converter. - The N9364C-GX role can be changed from the APIC to be a spine (default mode is Leaf). Once done, it needs to be rebooted. Once it comes back up, All 64 ports become Fabric ports (where you can connect leaves to). It is best not to connect leaves to ports 61 and 62. These can be converted to downlinks for IPN/ISN connectivity when doing Multipod/Multisite.
Joining N9364C-GX as a ACI Leaf
Given that port 63 and port 64 are the Fabric Ports for that leaf, for that node to join the fabric as a leaf, connect those 2 ports to existing spines and the node will get discovered as a leaf. You can then go to the APIC UI and register that node to the fabric (give it name and nodeID). The Leaf will show up as active and healthy.
Joining N9364C-GX as a ACI Spine
If you wanted the N9364C-GX to function as a Spine you would connect port 63/64 to existing leaf ports. The Node will get discovered as a Leaf. Remember in the ACI Fabric (Clos topology), Leaves connect to Spines. Leaves don’t connect to Leaves (generally), so this is an invalid topology.
- You would then have to go to APIC UI, Right Click on that Node and choose Spine as the role.
- Wait for a few minutes to ensure that the APIC has pushed the config to the node. Then reboot that Node. Once the Node comes up it will come up as a spine and be active.
- Go ahead and register the node to the fabric.
- This node will now function as a spine and you can connect any port on this spine to other leaves
📙 Remember that the last 2 ports are for Fabric Discovery purposes.
Adding N9364C-GX switch as a Spine during brand new/initial ACI Fabric deployment
This section is explicity the reason why I am writing this article. I got pulled in by a customer because they were having issues with this.
To start off their brand new ACI Fabric Deployment, their initial/starting Fabric topology is as shown below.
Figure 2: Initial/Starting point for building brand new ACI Fabric (for customer)
Problems Encountered:
- The first problem was that they had connected their N9364C-GX Spines to port E1/60 from the leaf. Since the N9364C-GX is by default a leaf switch with port 63 and 64 being the fabric discovery ports, the N9364C-GX was not getting discovered. After reading the Cisco documenation we connected the ports on the N9364C-GX ports 64 to the leaf as shown in the diagram above.
- Meanwile we also noticed that the customer had configured the infra VLAN to be vlan 1001 which is normally not a good idea (generally 3967 is a common good choice). For that reason we decided to clean up the configuration on the APIC and the registered leaf and start from scratch.
- Once the APIC rebooted, they did the intial config on it again (this time with Vlan 3967 as the Infra Vlan)
- After a while the Leaf showed up as discovered but it never went to active state. We waited for a long time and realzied that this was not working. On further analysis we realized that the connected N9364C-GX nodes were at their default leaf mode configuration. This was an invalid topology since Real Leaf was connected to 3 N9364C-GX which were supposed to be spines but were in default Leaf mode.
Analysis and Solution:
- For this brand new Fabric bringup, we disconnected the N9364C-GX Nodes (default Leaves).
- We then rebooted the real Leaf and the Leaf came up and became active in a few minutes.
- We then registered the Leaf to the ACI Fabric (from APIC UI)
- Once the first Leaf was registered and happy, we connected back the N9364C-GX (port 64) back to the leaf.
- From here the procedure was the same as adding N9364C-GX as spines to an exiting ACI Fabric. The N9364C-GX, got discovered as leaves. We changed to role of them to Spines from the APIC UI and rebooted them. They came back up again as spines and went active. We registered the Spines to the Fabric and all was done.
📙 all was done for bringing up the ACI Fabric. Customer now needs to do their initial best practice configurations, for fabric etc, etc, before onboarding tenants