The top question all new ACI customers have (or should have), is what are the configurations that should be enabled on my fabric from the beginning? With that in mind, we’re going use this post as a living document with configurations that are considered “Best Practice” to have enabled. We will keep this document updated as new versions come out, so don’t forget to bookmark this page! Wherever possible, we will include the Cisco documentation for the links, or at the very least, a detailed explanation of our reasoning.

Global Settings Best Practices:

1. MCP (per Vlan) should be enabled – MisCabling Protocol (or MCP) detects loops from external sources (i.e., misbehaving servers, external networking equipment running STP) and will err-disable the interface on which ACI receives its own packet.
   1. This can be enabled by going to Fabric > Access Policies > Global Policies > MCP Instance Policy default
   2. Make sure and enable the “Enable MCP PDU per VLAN” option (available after 2.0(2)), which enables MCP to send packets on a per-EPG basis, otherwise, these packets will only be sent on untagged EPGs (which basically makes it useless from a loop-detection perspective).
   3. If you want to read more about, MCP, go check out this post!
2. Disable Remote EP Learn – This will disable remote IP learning on border leaf switches.
   1. Prior to 3.0, this can be enabled by going to Fabric > Access Policies > Global Policies > Fabric Wide Setting Policy
   2. After 3.0, this can be enabled by going to System > System Settings > Fabric Wide Setting
   3. This is first available starting with 2.2(2e) and all code after
   4. Be aware of CSCvi11291 (fixed in 3.2(1l) and later). This bug will allow remote EP learns on border leaf switches even if Disable Remote EP learn is configured when the switch receives packets with src/dst of tcp 179.
3. Enforce Subnet Check (will only work on -EX and -FX based leafs)
   1. Prior to 3.0, this can be enabled by going to Fabric > Access Policies > Global Policies > Fabric Wide Setting Policy
   2. After 3.0, this can be enabled by going to System > System Settings > Fabric Wide Setting
   3. This is first available starting with 2.2(2q) and all 2.2(x) code after
      1. Not available in 2.3(x)
      2. First available for 3.0 starting with 3.0(2k) and after
   4. Enforce Subnet Check is somewhat like “Limit IP Learning to subnet”, but on steroids. You might remember that the “Limit IP Learning to subnet” BD configuration option prevents the learning of IP endpoints if they are not a subnet configured on the BD. “Limit IP Learning to subnet” does NOT drop the packet, it just stops it from being learning on the BD. The packet can still be learned on a leaf that does not have the BD configured (i.e., a border leaf). This can be problematic, and thus, the need for the Enforce Subnet configuration option. When enabled, we will not learn the IP component at the VRF level as well.
   5. Be aware of CSCvh17285 (fixed in 3.2(1l) and later). When Enforce Subnet is enabled, any Bridge Domains which are configured as L2-only -AND- have L2 Unknown unicast set to proxy will result in mac addresses not being learned from ARP/GARP packets. The workaround is to have the L2 BD configured for L2 Unknown Unicast = Flood.
4. EP Loop Detection
   1. While the EP Loop Detection configuration has good intentions, (i.e., finding a loop, and killing it), I have found that it is triggered as often (or more) by false positives, such as Vmotions of VMs, as it finds true loops. For this reason, I would disable EP Loop Detection.
   2. Prior to 3.0, this can be disabled (or enabled) by going to Fabric > Access Policies > Global Policies > EP Loop Detection Policy
   3. After 3.0, this can be disabled (or enabled)  by going to System > System Settings > Endpoint Controls > EP Loop Detection
5. IP Aging should be enabled
   1. When IP Aging is not enabled (which is the default), if multiple IP’s are learned on a single MAC, then as long as the MAC is active then all IP’s will stay learned on the fabric. Cosmetically, this is undesirable in scenarios where DHCP enabled hosts get a new IP address but both IP’s are still shown within the EPG operational tab as tied to that MAC.  This feature will age each IP separately to address that scenario. At 75% of the endpoint retention timer, a directed ARP is sent to the IP component of the endpoint, and if unanswered, ACI will allow the IP endpoint to age out.
   2. Prior to 3.0, this can be enabled by going to Fabric > Access Policies > Global Policies > IP Aging Policy
   3. After 3.0, this can be enabled by going to System > System Settings > Endpoint Control > IP Aging (look to the right for this tab)
   4. This is first available starting with 2.1(1h) and all code after
6. Rogue Endpoint Detection should be enabled.
   1. Starting with 3.2, Rogue Endpoint detection will lessen the impact from flapping endpoints.
   2. When Rogue Endpoint detection is enabled, the misbehaving endpoint (MAC/IP) will be quarantined and a fault will be generated to allow for easy identification.
   3. After 3.2, this can be enabled by going to System > System Settings > Endpoint Controls > Rogue EP Control
   4. Recommended Values:
      1. Rogue EP Detection Interval = 30
      2. Rogue EP Detection Multiplication Factor = 6
   5. NOTE – Rogue Endpoint Detection should be disabled prior to upgrading or downgrading the fabric. It can be re-enabled once the upgrade/downgrade is complete. This is documented in the APIC Basic Configuration Guide on CCO.
7. Enable Strict COOP Group Policy
   1. The APIC provides a managed object (fabric:SecurityToken), that includes an attribute to be used for the MD5 password. An attribute in this managed object, called “token”, is a string that changes every hour. COOP obtains the notification from the DME to update the password for ZMQ authentication. The attribute token value is not displayed.
      There are 2 choices,  Compatible Type and Strict Type.  Compatible Type accepts both MD5 authenticated and non-authenticated ZMQ connections, whereas Strict Type only allows MD5 authenticated ZMQ connections.
   2. This can be enabled by going to System > System Settings > COOP Group

At a high level, options 2 and 3 will prevent the mis-learning of IP endpoints on your fabric that can occur. Mis-learning of endpoints leads to things like black-holed packets, as a remote IP endpoints can get stuck on a border leaf (for example). The process of clearing such events is cumbersome and causes a lot of heartburn. For detailed examples of use cases for each of the endpoint configuration knobs, please check out the ACI Endpoint Learning Whitepaper (below). While I always recommend that these changes are performed in a maintenance window, the impact from enabling these options would be basically non-existent (i.e., a flush of remote IP endpoints in the VRF will occur). 

ACI Endpoint Learning WhitePaper

https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739989.html

Are you looking for a programmatic way of enabling all of the Global Setting Best Practices with a shell-script? Take a look at our ACI Best Practices for curling article!!

Bridge Domain Best Practices:

For Bridge Domains, there are a wide-mixture of use-cases, and lots of perfectly valid use-cases for different configurations. So – in general, best practice is in the eye of the beholder. However, with that being said, I’ll try a few blanket recommendations, with appropriate caveats.

1. Do not enable Unicast Routing if ACI is not the L3 Gateway for your Subnet.  Why would you ever enable unicast routing if ACI is not the L3 Gateway?  Without Unicast routing enabled, ACI will not learn the IP address for Endpoints. This leads some customers to enable Unicast routing, because (understandably) they want to learn the IP endpoint and not just the mac-address of connected devices. The problem with this, is it can lead to asymmetric routing, which can result in packets being dropped or mis-routed.
2. Configure a single subnet for each Bridge Domain. ACI will only forward dhcp requests on the primary subnet for each BD. If you have second subnet configured on the same BD, DHCP will not work for the 2nd BD and beyond.
3. In Network-Centric Mode (i.e., VLAN=EPG=BD), Do not configure multiple EPGs to a BD. When you mapping Vlans to EPGs and BDs in ACI, the external STP and HSRP multicasts are flooded in the same BD. For example, if you have Vlan 11 (EPG11) and Vlan12 (EPG12) attached to the same BD, HSRP hellos for both Vlans will intermingle in the BD and cause problems in your external (non-ACI) environment.
4. Enable Limit IP Learning to Subnet – This should be enabled 99.999% of the time. It limits the IP learning of endpoints based on the subnets configured on the respective bridge domains. Note – If you have -EX or -FX based leafs and have configured “Enforce Subnet Check” Globally, this is turned on whether you have enabled it or not. 🙂
5. Consider ARP Flooding + GARP-based detection – This is a 50/50 recommendation. I could go either way, but if it is my datacenter, I’m probably going to enable this configuration option. The Pro’s for GARP-based detection is that it will prevent IP learning issues in a specific situations. The Con is that you have to enabled ARP Flooding on the BD before you can configure the GARP-based detection. From the Cisco ACI Fabric Endpoint Learning Whitepaper – “Although Cisco ACI can detect MAC and IP address movement between leaf switch ports, leaf switches, bridge domains, and EPGs, it does not detect the movement of an IP address to a new MAC address if the new MAC address is from the same interface and same EPG as the old MAC address. When the GARP based detection option is enabled, Cisco ACI will trigger an endpoint move based on GARP packets if the move occurs on the same interface and same EPG. If a GARP packet comes from the same interface and same EPG, then endpoint learning is triggered only when Unicast Routing, ARP Flooding, and “GARP based detection” are all enabled for the bridge domain. Although this scenario has not been widely seen across our customer base, in some cases customers do change their IP to MAC bindings and need to enable GARP-based detection.”

Fabric Provisioning Best Practices

Performing an ACI Fabric Setup is one of the best things about ACI. However, proper planning for your fabric setup values is critical. When considering the values for your ACI fabric, it is important to remember that changing either the infrastructure IP address (TEP IP pool) range or the infra VLAN after the initial provisioning setup process is not possible without rebuilding the fabric.

When performing your initial Fabric Setup, you are required to input a “TEP address range”. This range of IP addresses is used primarily to provide TEP addresses for Leaf and Spine nodes in the fabric. While the default value for this is 10.0.0.0/16, it is considered best practice to provide a unique address block for your TEP pool for a couple of reasons:

1. If you want to extend your TEP pool to AVE (ACI Virtual Edge) switches in the future, you want a unique address that does not overlap with existing routing in your network.
2. If you want to have communication to external devices from the APIC (i.e., VCENTER for VMM integration), you would want addressing on your infra TEP pool that is unique to avoid IP address / routing conflicts for traffic coming back to the APIC from your VCENTER device.
3. Note – Changing the infrastructure IP address range or the VLAN after initial provisioning is not possible without rebuilding the fabric.

The Infra Subnet should not overlap with any other routed subnets in your network. If this subnet does overlap with another subnet, change this subnet to a different /16 subnet.

• Beginning with APIC 2.2 code, the minimum supported subnet for a 3-APIC cluster is a /23.
• If you are using APIC 2.0(1) code up until APIC 2.2 code, the minimum is /22.
• Infra TEP IP should be unused and unique. However, if you do not have any spare RFC1918 addresses, consider using the RFC6598 range (100.64/10  –  CGN use). This will ensure that this is never conflicted on the internet.
• Every Fabric / POD infra TEP pool should come from a unique IP subnet range.

For more information about this, check out the Cisco APIC Getting Started Guide, Release 3.x guide on CCO.

Infra Vlan ID – Set your Infra Vlan to 3967

During fabric setup, ACI requires a VLAN to be used as the infrastructure VLAN. This VLAN is used for control traffic between devices that make up the fabric (i.e., leafs, spines, and APICs).

Because this vlan can be extended outside of the fabric (Openstack integration, AVS/AVE), it is a best practice to have this as a unique Vlan in your environment. In addition, many Cisco devices have reserved Vlan ranges that are hard to modify (i.e., you have to reboot the switches for changes to take effect). Vlan 3967 is a Vlan which is not reserved on any Cisco switching platform and ideal for ACI.

Node ID Settings – Spines should be numbered between 101-199; Leafs should be numbered 200 and above.

For more detailed information, check out the Cisco ACI Best Practices Guide for Fabric Provisioning.

ACI Fabric Naming Best Practices

Need a good primer on ACI Fabric Naming best practices? Check out this post for suggested tips on naming your objects in both the Tenant and Fabric Access Section of your fabric!

Also – please check out the Official Cisco ACI Best Practices guide on CCO!