Using Standby APIC to move APIC or Replace APIC

Having a standby APIC is very useful in case of APIC failure, so you can promote the standby to take over the failed APIC.  

The benefits of having standby APIC is that during upgrade, the standby APIC upgrades automatically, so you always have a standby APIC in the correct version ready to go.

One overlooked feature of standby APIC is that it can be used to replace a APIC or move an APIC to a different location. For instance in a ACI Multipod Fabric, you can move a APIC easily from POD1 to POD2 using this method.  Or, maybe you are about to upgrade your APIC from APIC-L2 to APIC-L3 (or APIC-M3).  Using a standby APIC makes this really easy.

In this example I show  a multipod fabric with 2 PODs.  APIC1 and APIC2 are in POD1 and APIC3 is in POD2.   I want to move APIC 2 from POD1 to POD2.  I will setup a standyb APIC in POD2 and then switch over so that APIC2 is removed and the standby APIC now assumes APIC2 role and at the same time is moved to POD2.

Figure 1

Before Switchover, let’s take a look at the APIC GUI to see what the APICs currently look like.  This is shown below.

Figure 2

Let’s verify that all the APICs are synced up and “Fully Fit”

Figure 3

Let’s also click on Standby APIC tab to see if I have any standby apic currently

Figure 4

Since I don’t have any standby APIC, I will install one in POD2 and connect the uplinks (eth2-1 and eth2-2) to a pair of leaves in POD2 as shown in Figure 1.

A Quick Note on APIC M3/L3:

On APIC M3/L3 the default configuration from CIMC comes with hardware port Channel enabled.    Remember that the APIC does a active/standby bonding on the 2 uplinks to the leaves.   If you use eth2-1 and eth2-2 with this default configuration, your APIC’s LLDP neighborship will get really hosed and your APIC will never join the fabric.    You could either connect eth2-1 and eth2-3   or eth2-2 and eth2-4 to overcome this.  What I prefer doing is to go to CIMC and turn the default hardware Port Channel off.  Then I use eth2-1 and eth2-2 and everything is fine.    To do this go to CIMC of your APIC M3/L3, click on Networking/Adapter Card1, and then turn off Port Channel and save the config as shown in the diagrams below. ( a reboot of the host  will be required for this to take effect).

Figure 4a

First, gather all information for the APIC cluster setup so we know what to put in the standby APIC initial setup.   You can do this via GUI or from CLI on APIC.

GUI Method:

Figure 5

CLI Method (ssh to any apic):

cat /data/data_admin/sam_exported.config

Figure 6

Second, I would suggest to upgrade the APIC manually to the current version that you are running in production, without connecting it to the fabric.  This can be easily done through GUI or manually.  After the upgrade connect up the uplinks to a pair of leaves.

Third, go in through CIMC console and setup the standby APIC.   Use the information gathered above to populate the startup wizard.  Also make sure of the following.

  • put the correct POD ID where the APIC will be moved
  • make sure you select “standby”
  • make sure to name the apic with the name that the new apic will have
Figure 7
figure 8

After a few minutes ping the OOB IP to make sure that it’s reachable

Figure 9

From the console you will see that standby apic is on login screen

Figure 10

You will need to login as rescue-user with no password if you want to look around.  (admin password will not work)

Figure 11

You could also ssh in as “rescue-user”  using the oob IP of standby APIC

Some Basic commands will work from here.  Some are shown below.

Figure 12
Figure 13

APIC GUI should now show the standby APIC with the serial number

Figure 14

On GUI go to APIC/Cluster as seen by Node and click on “Standby APIC”

Figure 15

You will now see the new Standby APIC there

Figure 16

Now, let’s replace APIC 2 with the standby.  

From GUI:

  • first make sure that you go to any other APIC other than the one you are replacing. 
  • Click on Cluster as seen by Node 
  • Click on the APIC you want to replace and click the right mouse button and click Replace
Figure 17

The Default is to Retain OOB IP for Standby

Figure 18

From GUI you will see that the old APIC2 is now unavailable and is in “reprovisioning” state

Figure 19

The Original APIC will be automatically shutdown

Figure 20

You will also notice that Standby APIC is gone now

Figure 21

The original APIC will then show Unregistered and waiting for new APIC

Figure 22

On the Console of Standby APIC you will notice that the Standby APIC will reboot

Figure 23

On the GUI you will see that the APIC has now come in and is syncing the database

Figure 24

In a few minutes, it should be done.  You will notice that the APIC ID is now the same as the original APIC that you replaced

Figure 25

You can check the IP and see that it’s retained the new Standby APIC IP

Figure 26

You are all done !!! 

Make sure to disconnect the old APIC.    SSH to it and clean it up with the following commands:

  • acidiag touch clean
  • acidiag touch setup
  • acidiag reboot

Conclusion:

  • The Process is really simple and quite fast.
  • Though you could easily do this without maintenance window, it’s better to do it during a maintenance window

References:

Cisco APIC Getting Started Guide, Release 4.2(x)
Posted in All

6 thoughts on “Using Standby APIC to move APIC or Replace APIC

    1. Thank You Roman. I spend a lot of effort and time in writing these articles. Your comment makes it all worth while !

    1. Thank You so much for the great feedback ! I spend a lot of my own time writing these articles and this sort of feedback makes it worth my time and effort.

    1. Thank You so much Stijn ! I really appreciate your feedback and knowing that these are helping the community gives me incentive to keep going !

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.