Upgrading ACI Fabric: The 4.0 Way.

Now that 4.0 is released, it’s time to start planning your upgrade! So get out your pencils and slide rules, and start blocking off your calendar for maintenance windows for the next three months…

Just kidding! With ACI, upgrading is simply one of those casual tasks you do between coffee breaks or network outages. And you don’t even need the slide rule. That’s for the old-school network guys who like manually updating firmware switch-by-switch…and that doesn’t describe you, of course.

Important Notes:

This guide is designed around the upgrade process in the 4.0 interface.

If you’re still running 3.x or earlier and you’re not upgrading to 4.0, you’ll actually want to follow this guide, instead.

If you’re running 3.x and upgrading to 4.x, the GUI/web interface will change mid-upgrade. Use these instructions to upgrade the controllers, and then come back here and jump to Step 2 under Upgrade the Fabric below for the switches themselves.

Two important differences between the 3.x and 4.x interfaces:

• Firmware Groups and Maintenance Groups are combined into a single Upgrade Group, which can be created during the upgrade process, rather than as separate steps. This can be found @ Admin > Firmware > Infrastructure > Nodes
• Enforce Bootscript Version Validation been moved from the Fabric Node Firmware screen to the Admin > Firmware > Infrastructure > Nodes screen.

Obligatory Pre-work (Read This or Crash and Burn)

1. Create a backup of your ACI Fabric prior to upgrading. This should be a no-brainer, but is often overlooked. I always recommend generating a backup of your ACI Fabric configuration both BEFORE and AFTER an ACI Fabric upgrade. The rule of thumb is to always have a backup based on your current running version of code. Need help with creating a backup? Just go to the Creating a backup for your APIC Cluster post.
2. Always make sure that all of APICs are in a Fully Fit state prior to any upgrades. Never kick-off an upgrade of your fabric unless all APICs are in a Fully Fit state!
3. Resolve faults prior to your upgrade. This is often overlooked, but can lead to serious issues. It is always recommended to have as many faults resolved prior to upgrading the fabric.
4. Review the Cisco CCO APIC Upgrade/Downgrade Guide. This guide is comprehensive and covers which versions you can upgrade/downgrade to and from, guidelines, and suggested maintenance group configuration.
5. Determine which version of code if best for you and your ACI Fabric. If you have questions on what version of code you should target, check out post on ACI / APIC Software Guidance for a quick reference cheat guide.
6. Check out Recommended APIC Code page on CCO; great place to find out the long-lived ACI releases and supported CIMC images.
7. Check Release Notes for your intended version. It’s always good to check the release notes for any bugs, limitations, or version specific notes BEFORE you pull the upgrade trigger.
   • APIC 2.2(4) Release Notes
   • APIC 3.2(3) Release Notes
   • APIC 4.0(1) Release Notes
   • APIC 4.1(1) Release Notes
8. Download the code from CCO. You’ll need two files; the ISO file for your APIC Cluster, and the BIN file for your switches (one BIN file will work for both Spines and Leafs).
9. Check your CIMC code and ensure you are running supported code on your CIMC. If you need help with upgrading your CIMCs, check out the Upgrading your CIMC for your APIC article here.

Upgrading your Fabric: The Process

Now that the pre-work is complete, lets move onto the actual upgrade!

Copy the ISO and BIN files to the APIC

There are several ways to get files from CCO onto your APIC Fabric.

1. SCP the files to the APIC
2. Upload the files via the GUI

While you can upload from the GUI, I prefer to use SCP to transfer the files to the APIC. I personally like to see the byte counts offered via SCP as opposed to the ominous transfer bar in the GUI 🙂

Option 1 – SCP the files to your APIC. 

• You will notice a couple of things about the scp command. “admin” is sample username we are using for the APIC.
• The IP address field is the IP of one of your APICs from your APIC Cluster (you only need to upload the files to one APIC).
• Don’t forget the colon “:” between the IP/ServerName and the directory
• New to SCP? Check out this helpful link.

1. Copy ISO file for APICs – scp filename.iso admin@10.18.188.101:/tmp/.
2. Copy BIN file for Switches – scp filename.bin admin@10.18.188.101:/tmp/. 

Add the files to the APIC Firmware Repository

1. Login to APIC and add firmware to firmware repo
   1. SSH to the APIC
   2. apic1# bash
   3. apic1# cd /tmp
   4. apic1# firmware add filename.ISO << go get a coffee. This will take 3-5 mins
   5. apic1# firmware add filename.BIN << stir your coffee. This will take 1-2 mins

Option 2 – Upload the files via the GUI

1. Upload ISO file for APICs
2. Upload BIN file for Switches

Go to Admin > Firmware > Images > Actions Button > Add Firmware to APIC

Verify the Firmware is present in the APIC Firmware Repository

1. Verify Firmware is in Firmware Repository on the APIC
   • Admin > Firmware > Images

Upgrade the Fabric!

Upgrade the APIC Controllers

• Admin > Firmware > Infrastructure -> Controllers
• Click Action button > Schedule Controller Upgrade

Set target firmware version and Select Upgrade Now Option
If you have any warnings, such as Major Faults, you must resolve these before upgrading!

• Set target firmware version and Select Upgrade Now Option
  If you have any warnings, such as Major Faults, you must resolve these before upgrading!

Set target firmware version and Select Upgrade Now Option
If you have any warnings, such as Major Faults, you must resolve these before upgrading!

• Click Submit
• Go refill your coffee, tell all your colleagues how hard upgrades are, and hide somewhere playing video games on your phone.

Upgrade your ACI Fabric Switches

• Go to Admin -> Firmware -> Infrastructure -> Nodes
• Select Action -> Schedule Node Upgrade

• 
  

   

  1. 1. You can use an existing upgrade group, or create a new one.
     2. If you are creating a new upgrade group, you’ll need to enter the node IDs for that group.
        • Select Range to specify a range of IDs separated by commas (eg, “101,201-209,301-303,402”). See the Upgrade Groups section below for ideas.
        • Use Manual to select checkboxes from a list of switches.
        • If you selected an existing upgrade group, the node IDs are filled in for you.
     3. Select your newly uploaded firmware version
     4. Choose your run-mode: “pause on failure” or “do not pause on failure”
     5. Select Now, unless you need to schedule a maintenance window.
  2. Refill your coffee cup, and let everyone know how hard you’re working on this upgrade.
• Use the newfound time savings to put proper cover sheets on your TPS reports.

Upgrade Groups

1. 1. Two-group method
      1. Divide your switches into two groups: a red group and a blue group. Put one half of the spine switches in the red group, and the other half in the blue group. Also, put one half of the leaf switches in the red group and the other half in the blue group.
      2. Upgrade the red group.
      3. After the red group upgrade is complete, confirm that the fabric is healthy.
      4. Upgrade the blue group.
   2. Four-group method
      1. Divide your switches into four groups: a red spine switches group, a blue spine switches group, a red leaf switches group, and a blue leaf switches group. Put one half of the spine switches in the red spine switches group, and the other half of the spine switches in the blue spine switches group. Then, place half the leaf switches in the red leaf switches group, and the other half in the blue leaf switches group
      2. Upgrade the red leaf switches group.
      3. After the red leaf switches group upgrade is complete, confirm that the fabric is healthy.
      4. Upgrade the blue leaf switches group.
      5. After the blue leaf switches group upgrade is complete, confirm that the fabric is healthy.
      6. Upgrade the red spine switches group.
      7. After the red spine switches group upgrade is complete, confirm that the fabric is healthy.
      8. Upgrade the blue spine switches group.

ACI Upgrade notables

• In an ACI Multi-Pod environment, it is recommended to have separate maintenance groups per Pod
  • Switches in different pods can only be upgraded serially, regardless if they are in the same maintenance group. The switches cannot be upgraded in parallel.
• For your BGP-RRs (your Spines) in each Pod, make sure you always have one BGP-RR available during the upgrade process. In other words, don’t place both all of your BGP-RR spines in the same maintenance group. This is especially true when performing an ACI Fabric Upgrade for Multi-Pod deployments.

How long will the upgrade take?

While I can’t tell you exactly how long the upgrade will take, there are some guidelines that can be provided based on experience. The following times are approximate.

• CIMC = 45-60 minutes per CIMC/APIC << (may not be required)
• APICs = 30-45 minutes for the APIC cluster
• Switch Maintenance Groups = 15-30 minutes per Upgrade Group

So, if you have an APIC cluster of (3) APICs, and four maintenance groups in a single pod ACI Fabric, you’ll be looking anywhere from 1hr 30 minutes to 2hrs 45 minutes.