ACI / APIC Software Guidance

Screen Shot 2018-04-03 at 2.58.16 PM.png

DISCLAIMER – This post is not meant to take the place of a software recommendation process.  Customers should still perform their own due diligence before selecting a software version for their own ACI Fabric. This post is meant to summarize the available ACI Software features, by release, and be used as a quick reference guide. I’ll offer up my favorite version as well, but as all good consultants know, when asked “what version should I run”, the answer is always “It Depends”. 😉

It depends. As someone who has been in a consulting role for a long time, this is my go-to answer. Although I do use this statement whenever I am in doubt of an answer, I just as often use It depends even if I feel I know the answer. This is especially true when it comes to providing Software Guidance for ACI. While I have a preferred “go-to” software version that I start with, a lot depends on the customer, the features they are running in their environment, the hardware, and where they are going in the near future. Take a look at the versions below. I’ve tried to keep it as straight forward as possible, including where Hardware and Software features were first supported. I’ve also included a few versions never to run, based on my own experience.

My Guidance – I would start with the latest gold star release in the 4.2(x) train. Here are my reasons why:
  • It is a Long-lived Supported APIC/ACI Code Release. Cisco supports direct upgrades from long-lived release to long-lived release, which means you don’t have to step upgrade to get to the previous (or next) long-lived release! If you want to verify what the upgrade path looks like, check out the ACI Upgrade Tool on CCO to verify.
  • Lots of features!
    • MultiPod Support
    • MultiSite Support
      • Lots of MultiSite enhancements!
    • MultiPod + MultiSite
    • Lots of ACIAnywhere enhancements (AWS + Azure)
    • Remote Leaf Support
    • Cisco AVE
    • Flood in encap (critical for ACI-Centric deployments)
    • Vmware Vsphere 7.0 VMM is supported
    • Application Services Engine Support
Now that I’ve given you a starting point, take a look at the table below and determine what works best for you and your ACI Fabric.

APIC 1.0

1.X (any 1.x release) – This version is end of life. If you are running a 1.X release of any kind, you should consider moving to the 3.2 release. Check out the link below to help determine which versions you can upgrade to and from:

In addition, here are the end-of-life announcements for all 1.x APIC Software.

Do you need help upgrading your Fabric? Check out this post on Upgrading your ACI Fabric.

1.3 (Bronx)

  • New Hardware Supported
    • N9K-C93180YC-EX (Leaf)
    • N9K-X9732C-EX (Spine LC)
    • N9K-C9504-FM-E (Spine Fabric Module)
    • N9K-C9508-FM-E (Spine Fabric Module)

APIC – 2.0

2.0(2) (Congo)

  • New Software Features
    • VMM – VMware VCenter 6.0 is supported
    • Contract Permit Logging
    • MultiPod – MultiPod support introduced
    • Copy Services
    • EPG deployment via AAEP
    • L3 multicast support (requires at least -EX based Leaf)
    • Policy-based redirect
    • Syslog in NXOS Style CLI Format
    • Proxy ARP
    • Per-EPG MCP
  • New Hardware Supported
    • N9K-C93108TC-EX

2.1 (Crystal)

  • New Software Features
    • MultiPod – Copy Services Support for MultiPod
    • MultiPod – Golf support for -EX based switches
    • FIPs Support
    • IP Aging (Endpoint Learning Best Practice)
  • New Hardware Supported
    • QSA support for N9500 Spine Linecards and -EX-based Leafs

2.2 (Danube)

  • Guidance – Latest 4.2(x) is preferred.
  • Endpoint Learning Bug – CSCvi11291 – XR learn on BL even with “Disabled remote EP learn” for BGP packets (tcp port 179). This bug is first fixed in 2.2(4m) and 3.2(1). This issue is most commonly seen when you have external security port-scanners (or other devices generating tcp-179 packets) that are sent to endpoints on the ACI fabric. When this occurs, Remote (XR) Stale entries can pop up. This issue is resolved in 2.2(4m) and 3.2(1).
  • You can upgrade directly from 2.2(4) to 3.2(2) (which is the next long-lived release for ACI). For more information on long-lived releases, check out this link on CCO.
  • New Software Features
    • VMM – VMware VCenter 6.5 is supported
    • Critical Best Practice Endpoint Learning Options are available
      • Enforce Subnet Check
      • Disable Remote EP Learning
    • MultiPod – Active/Standby FW support across Pods (MultiPod) without vPC (physical link or local port-channel only)
    • Cisco ACI App Center
    • Standby APIC
    • Contract Preferred Groups
    • Netflow for -EX based Leaf Switches
    • Control Plane MTU Setting (for use with MultiPod)
    • Q-in-Q Tunneling BD
  • New Hardware Supported
    • N9K-93180LC-EX (40Gig EX-based Leaf)
    • Breakout support for 9332
    • N9K-C93180YC-FX – 2.2(2)
    • N9K-C93108TC-FX – 2.2(2)

2.3 (Drava)

  • Not a long-lived train; Enforce Subnet Check (Endpoint Learning BP) is not available for any 2.3 code version.
  • If needed, use latest 2.3 release on CCO.
  • New Software Features
    • MultiPod – Active/Standby FW support across Pods (MultiPod) with vPC
    • Attribute based uSeg (Microsegmentation)
    • Contract Inheritance
    • Tetration Analytics support for FX-based Leaf Switches
  • New Hardware Supported
    • N9K-SUP-A+, N9K-SUP-B+

APIC 3.0

3.0 (Ebro)

  • Do not use 3.0(1k) or 3.0(2h) due to CSCvg38918 – DHCPv6 related memory leak (you do not have to have IPv6 enabled on the fabric for this issue to affect you!)
  • Other bugs to be aware of:
    • Endpoint Learning Bug – CSCvi11291 – XR learn on BL even with “Disabled remote EP learn” for BGP packets (tcp port 179). This bug is first fixed in 3.2(1). This issue is most commonly seen when you have external security port-scanners (or other devices generating tcp-179 packets) that are sent to endpoints on the ACI fabric. When this occurs, Remote (XR) Stale entries can pop up. This issue is resolved in 2.2(4m) and 3.2(1).
  • New Software Features
    • MultiSite – MultiSite is first supported; (N9K-X9732C-EX Spine Linecards are required)
    • VMM – Kubernetes for bare-metal server support
    • Intra-EPG Contracts
    • Tetration Analytics support for N9K-C9348GC-FXP switch
  • New Hardware Supported
    • N9K-9364C (no MultiSite support for 3.0)
    • N9K-C9348GC-FXP (1 RU, fixed port 48port 10/100/1000)
    • N9K-C9508-FM-E2 (Spine FM)
    • N9K-C9736-FX Spine LC

3.1 (Euphrates)

  • Do not use 3.1(1i) due to CSCvh29461 – DSCP-cos translation policy may break MultiPod BGP. This issue is resolved in 2.2(4f) and later in the 2.2(4) train, and in 3.1(2m) and later in the 3.x train.
  • Other bugs to be aware of:
    • Endpoint Learning Bug – CSCvi11291 – XR learn on BL even with “Disabled remote EP learn” for BGP packets (tcp port 179). This bug is first fixed in 3.2(1). This issue is most commonly seen when you have external security port-scanners (or other devices generating tcp-179 packets) that are sent to endpoints on the ACI fabric. When this occurs, Remote (XR) Stale entries can pop up. This issue is resolved in 2.2(4m) and 3.2(1).
  • New Software Features
    • Monitor Active GUI Sessions
    • BFD support for Spine switches
    • Cisco AVE (Next-Gen AVS)
    • L4-7 Cloud Orchestrator Mode
    • Flooding is limited to Encapsulation (Flood-in-Encap)
    • Downlink support for Uplink ports on EX-based and FX-based Leaf switches.
    • OpenShift Container support
    • Remote Leaf Switches
    • MultiSite – N9K-C9364C Spine Switch is now available for MultiSite
  • New Hardware Supported
    • Beginning with ACI 3.1(2) – N9K-C9336C-FX2

3.2 (Fraser)

  • Guidance – While Latest 4.2(x) is preferred, 3.2 is also a current long-lived release. If you don’t require any of the features in 4.x train, please use the gold star release for 3.2.x
  • If needed, do not use 3.2(1l) due to CSCvj65274 – Switch crash possible during upgrade to 3.2(1). Switches with Call Home Inventory Policies enabled and applied to switches may encounter a switch crash with the eventmgr service.
  • Do not use 3.2(5d) due to CSCvo83991 – SNMP traps not being sent from the TOR. 
  • All versions of ACI code up until 3.2(6i) are vulnerable to CSCvo80686 – Cisco Nexus 9000 Series Fabric Switches ACI Mode Default SSH Key Vulnerability.
  • Other bugs/enhancements to be aware of:
    • CSCvm12554 – Contract Preferred group l3out prefix not deployed on ingress VPC; this bug was re-introduced to 3.2 affects all available versions of 3.2 up until 3.2(4d) where it has been fixed again.
    • Endpoint Learning bug – CSCvi11291 – Remote Learn on Border Leaf even with Disabled Remote EP learn with pkt with src/dst of 179. This issue is most commonly seen when you have external security port-scanners (or other devices generating tcp-179 packets) that are sent to endpoints on the ACI fabric. When this occurs, Remote (XR) Stale entries can pop up. This bug is first fixed in 3.2(1).
    • Endpoint Learning Enhancement – CSCvj17665 – EP announce support for stale IP XR EPs – This enhancement improves endpoint learning functionality by allowing a new EP Announce delete message to be sent to all leafs within the site on the expiration of Bounce IP XR Entries. This enhancement is available beginning with 3.2(2).
    • CSCvj90443 – Preconfigured VPC can lead to duplicate VIP/TEP IP assignment – This issue is resolved in 3.2(2o) and later
  • 3.2 is a long-lived code train starting with 3.2(2); see the link here.
  • New Software Features
    • Layer-3 routed and sub-interface port-channel for L3out
    • SPAN on L3out
    • Multi-Site + Multi-Pod Support
    • Multi-Site Back-to-Back Spine
    • VMM – VMware VCenter 6.7 is supported beginning with 3.2(2)
    • MCP Aggressive timer support
    • Remote Leaf – Orphan Port support
    • UI Enhancements
    • Fibre Channel N-port virtualization
    • Rogue Endpoint Control Policy
  • New Hardware Supported
    • Enhanced breakout support on profiled QSFP ports on N9KC93180YC-FX switches
    • The Cisco N9K-C9336C-FX2 switch now supports breakout, 18-port downlink/uplink, and MACsec.

APIC 4.0

4.0 (Ganga)

  • New Software Features
    • EPG Shutdown
    • Disable IP Dataplane Learning (VRF Level)
    • Multi-Site – L4-7 Service integration
    • Multi-Site – CloudSec
    • Multi-Site – L3 Multicast
    • RP in the Fabric
    • QOS for ROCEv2
    • Additional QOS classes (3 additional levels)
    • MACsec encryption support on remote leaf switches
    • TCAM Policy Compression for identical filter rules
    • Preferred Group support for service-groups
    • Inter-VRF Multicast
    • ACI vPOD (limited availability)
    • ACI Host-based Routing advertisement via L3out
    • L3out Supported in service-graphs
    • Fabric-wide CPU, memory utilization and temperature dashboard
    • VMM read-only domain promotion to fully managed
    • AVE Uplink VxLAN Load-balancing
    • Fibre-Channel enhancements
      • FCoE enhancements
        • vPC with SAN boot
        • vFC ports can now be a member of a vPC
      • NPV support enhancements
        • NPIV mode support
          • Host – 4G/16G/32G/Auto speed options
          • Uplink – 4G/8G/16G/32G/Auto speed options
          • Port-channel support on FC uplink ports
          • Trunking support on FC uplinks ports
  • New Hardware Supported
    • Mini ACI
    • Virtual APIC (vAPIC)
    • Cisco APIC-X
    • N9K-C9332C – 32 port 40/100G (Baby Spine)
    • N9K-C93240YC-FX2 – 48 port – 10/25G + 12 40/100G uplink ports

4.1 (Hudson)

  • New Software Features
    • ACI Anywhere with AWS
    • OpenStack OSP13 Support
    • MLD Snooping
    • GTP Load-balancing
    • L1/L2 PBR
    • Multi-Tier Topology support
    • UCS Integration App
    • Cloud APIC
    • Cisco ACI Integration with Cisco’s SD-WAN Viptela
    • Support for Microsoft NLB
  • New Hardware Supported
    • Nexus N9K-9358GY-FXP
    • Nexus N9K-X9736Q-FX
  • Scale Enhancements
    • Remote Leaf – 128 (Single Pod)
    • 100 sub-interfaces per VRF and Per L3out
    • 30K IPv4/IPv6 LPM prefixes on Border Leaf (EX, FX, and FX2 platforms)
    • 4K Mac EPGs
    • 32K L2 Multicast support on FX platform

4.2 (Indus)

Warning – Do NOT use 4.2(1i). Instead, use the latest gold star version of 4.2(x).

  • 4.2(x) – Long Lived Release and General Recommendation for customers (unless new features warrant newer version). Please use the gold star version in the 4.2.x train.
  • New Software Features
    • Floating L3out
    • Viptela SD-WAN integration
    • Backup (N+M) PBR
    • Cluster validation tool
      • ifav40-ifc1# acidiag cluster
    • PIMv6 support
    • ACI CNI SNAT
    • Docker Enterprise Edition Integration
    • Pre-download of Image for faster upgrade
    • CDP/LLDP support for management interfaces
    • 4.2(3) – Separating CRC errors and FCS errors
    • Multi-Pod Convergence improvements for L2, L3 unicast, and BUM cases
    • L3out Enhancements
      • BGP Session Shutdown and soft reset
      • IPv6 Multicast address family for BGP L3out
      • etc
    • Cloud Features
      • ACI Anywhere – Azure Extension
      • Ability to have cloud only deployments (AWS and Azure)
      • Support for CSR version 16.12
    • Multi-Site Enhancements
      • MSO Inter-version validation
      • Intersite L3out
      • AVE support (requires MSO 2.2(1))
      • Migration of EPG/BD
      • DHCP Relay for stretched BDs
      • 4.2(4) – vzAny contract support for Multisite
    • Remote Leaf Enhancements
      • 4.2(4) – Remote leaf support with 10Mbps link in IPN
      • 4.2(4) – Dot1q tunnel support on Remote Leaf
    • 4.2(4) – Allow custom names for ACI created port-groups (DVS, Hyper-V)
  • Scale Enhancements
    • 64 Remote Leaf nodes for RL Direct
    • 256K support of contract policy rules on FX TORs
    • SPAN/ERSPAN SCALE: 32 session scale & port scale increase to 63
    • MultiPod
      • 4.2(4) – Support for 500 leafs per Pod.
    • MultiSite
      • 4.2(4) – Increase preferred group scale to support 500 EPGs in a multisite deployment
      • 4.2(4) – Scale increase of up to 10 templates per schema from 5 templates per schema
      • 4.2(4) – Support for 500 leafs per site
  • New Hardware Supported
    • 4.2(2) – Spine: 16x400G ports spine : (N9K-C9316D-GX)
    • 4.2(2) – Leaf: 28×40/100G+8x400G ToR :  (N9K-C93600CD-GX)
    • 4.2(2) – RL POD Redundancy support
    • 4.2(3) – N9K-C9364C-GX support as ACI leaf
    • 4.2(3) – Application Services Engine
    • 4.2(3i) – N9K-C9364C-GX software support added

APIC 5.0

5.0 (Jordan)

  • New Software Features
    • Cloud Features
      • AWS Transit Gateway Support
      • AWS TGW Peering
      • Stats Filter for AWS
      • Cloud APIC UI topology view support
      • Common Driver for AWS and Azure
      • Cloud + CSR Upgrade Reconciliation
      • Cloud APIC Stateless cPE
      • AWS – 17.1 CSR Upgrade
    • Service Provider Features
      • SR-MPLS Features
        • SR-MPLS Handoff Support on ACI Border Leaf
        • QoS Marking for North-South and East-West traffic flow
        • SR-MPLS Statistics
      • Legacy SP PBR gaps
        • MAC rewrite
        • L3Out service EPG
      • PBR : L4-L7 devices(Deployed Device) in different BDs for load-balancing L1/L2 PBR
      • ACI Peering with vRouters (Floating L3out) a.k.a BGP next-hop unchanged propagation
    • Endpoint Security Group support
    • Usability & Day 2 Operations
      • During upgrade –> keep visibility on what’s happening on the APIC that is upgrading
      • MSO UI/UX enhancements for cAPIC
      • SNMP/Syslog apply policy in Day0 wizzard
      • Pre-download image to all/selected leafs
      • Trigger MPOD parallel switch upgrade across PODs
      • Upgrade Checker
      • ACI App Center RBAC
      • Two Factor Authentication with DUO for APIC, cAPIC
    • Openstack/Containers Features
      • Support for additional platforms
        • Openstack Ironic on Redhat OSP13
        • Kubernetes on DockerEE Release 3
        • Redhat Openshift 4.3 on AWS with ACI-CNI
        • Redhat Openshift 4.3 on OSP13 with ACI-CNI
      • Infrastructure improvements
        • Infra-VLAN hardening for Opensource VMMs
        • Restricted admin account for orchestrators like Openstack
        • Kubernetes support on mixed baremetal and VM topologies
        • Installer improvements (POD BD in common tenant,  validations, etc)
      • Data plane improvements
        • Add active/active VIP support on Neutron L2
        • Support SVI connectivity for OVS/opflex ports in Openstack
        • Improve scale per leaf pair  to support 120 Opensource OpFlex hosts
        • Support VMware teaming with enhanced LAG for Kubernetes nested in ESX VMM
  • Scale Enhancements
    • VMM Integration: Support to Increase the # of DataCenters in a vCenter Server from 4 to 15
    • Cisco N9K-C9336C-FX2 switch, you can now apply a breakout configuration on ports 1 through 34, which can give up to 136 (34*4) server or downlink ports.
  • New Hardware Supported
    • Cisco Nexus 9508-FM-G Fabric Module support

7 thoughts on “ACI / APIC Software Guidance

  1. Hey Jody, I see you’ve been writing a lot of cool articles in this blog lately, super useful in every possible way. Please keep it going 🙂

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.