Cisco MSO Standalone Upgrade

Are you using MSO (ACI Multisite Orchestrator), in standalone mode ?   Well, please do not do that in production.  Often times in a lab environment you will be tempted to do this.  I’m guilty of doing this myself on the Cisco  ACI Advanced Bootcamp ACI training labs, simply because I’m too lazy to bring up another 2 VMs and also to save on compute resources. 

So, now what I want to discuss here is how to upgrade the standalone MSO to a newer release.  Say you upgraded APIC+Fabric from 4.1.1i to 4.2.2e.  So, you want to upgrade your Standalone MSO from 2.1.1h to 2.2.2b.   I used to do this earlier easily but for the last 2 releases doing a standalone MSO upgrade has not been successful.  I internally looked for TAC cases on this issue and noticed that there were quite a few TAC cases opened on exactly this issue and for almost all these open cases, customer ultimately just installed  brand new standalone MSO and restored the backup from the previous MSO (if they took a backup from the previous MSO, that is).  Often times in the lab situation I am guilty of not doing a backup before upgrading.

So, what’s the error message you get when the upgrade bombs ?   Unfortunately, I don’t have a live capture right now, because I had to do this for a few MSOs in a hurry because students were waiting on me.  However the error is something to the effect that it tries to do a docker upgrade and then bombs and the docker engine is shut down and the upgrade fails saying something to the effect of   “is docker running ?”

 

If this happens, go ahead and start back docker back again.

systemctl start docker

systemctl enable docker

systemctl status docker

docker   –info

 

Here is the hack to get it to work:

Optional Steps:
Do a docker upgrade manually. 

If you opted to do this optional step, then make sure that you have your proxies configured in the Docker VM (CentOS).   Assuming your Internet connectivity needs a proxy.

first ssh as root to the MSO (obviously 🙂 )

shell proxy would look something like this:

/etc/profile.d/proxy.sh:
export http_proxy=http://mylabproxy.company.com:80/
export https_proxy=http://mylabproxy.company.com:80/
export ftp_proxy=http://mylabproxy.company.com:80/
export no_proxy=.mydomain.com
export HTTP_PROXY=http://mylabproxy.company.com:80/
export HTTPS_PROXY=http://mylabproxy.company.com:80/
export FTP_PROXY=http://mylabproxy.company.com:80/

Browser Proxy would look something like this:

/etc/envoronment

export http_proxy=http://mylabproxy.company.com:80/
export https_proxy=http://mylabproxy.company.com:80/
export ftp_proxy=http://mylabproxy.company.com:80/
export no_proxy=.mydomain.com
export HTTP_PROXY=http://mylabproxy.company.com:80/
export HTTPS_PROXY=http://mylabproxy.company.com:80/
export FTP_PROXY=http://mylabproxy.company.com:80/

YUM Proy would look something like this:

/etc/yum.conf

[main]
proxy=http:mylabproxy.company.com:80/

Docker Proxy:

mkdir /etc/systemd/system/docker.service.d/
sudo vi /etc/systemd/system/docker.service.d/http-proxy.conf

[Service]
Environment=”HTTP_PROXY=http://mylabproxy.company.com:80/”
Environment=”HTTPS_PROXY=http://mylabproxy.company.com:80/”

How to upgrade Docker.

yum check update
yum update -y
yum remove docker docker-common docker-selinux docker-engine
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager –add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install docker-ce
sudo systemctl start docker
sudo systemctl status docker
sudo systemctl enable docker
docker info

Required (non-optional)

1) scp the code to MSO, to directory /opt/cisco/msc/builds

e.g.

scp msc_2.2.2b.tzr.gz    root@msc_ip:/opt/cisco/msc/builds/

2) cd to directory /opt/cisco/msc/builds 

3) Untar,   tar -xvzf msc_2.2.2b.tzr.gz 

4) go to the prod-standalone directory for your release (from the untarred file)

cd /opt/cisco/msc/builds/msc_2.2.2b/prod-standalone

5)  vi the upgrade.sh file,  Search for  “upgradeDocker” and comment it out.

6) run the upgrade script —      ./upgrade.sh

The script will take some time to run ( I think 10 to 12 minutes)  and upgrade the MSO to the new release.

Check to make sure that docker swarm shows that all containers are up and replicated by using the command:  “docker service ls

O.K. All Looks good. You are done !

Log into MSO UI, make sure it’s working. After that go back to the SSH session. Use the command: “docker images” and you will see the old images there also.

Clean up the old images by using the following command:

docker system prune -a

Below screen-shot shows, I have reclaimed 10.5GB of disk ( I had more older images also than what’s shown in above screenshot). Now, the only images remaining are for 2.2.2b


3 thoughts on “Cisco MSO Standalone Upgrade

  1. Any recommendations on how to do an upgrade if using a 3 vm cluster for the MSO? The tools file includes a readme which I’ve followed, but it has been unsuccessful.

    1. Hi Mark,

      a) make sure to read the release notes to make sure that the upgrade path is compatible
      b) Please make sure to download a backup copy of the MSO config
      c) not really relevant, but please also see the “Deploying MSO on Cisco Application Service Engine OVa” I think you will really like that MSO.

      Now, going back to your question. The way, I’ve always done it with success is:
      a) scp the new code to all 3 MSOs to /opt/cisco/msc/builds. Extract the tar file (tar -xvf filename.tar or decompress).
      b) cd to /opt/cisco/msc/builds//bin/upgrade
      c) once there do a ls and you will see a readme.txt file.
      d) If you follow the instructions there, it should work.

      as an example:

      [root@node1 upgrade]# pwd
      /opt/cisco/msc/builds/msc_2.2.2b/upgrade

      [root@node1 upgrade]# ls
      readme.txt upgrade.sh

      [root@node1 upgrade]# cat readme.txt
      Multi-Site Orchestrator to upgrade

      1. Download and extract .tar.gz file on all 3 nodes
      a. download
      .tar.gz file into /opt/cisco/msc/builds/
      b. decompress
      .tar.gz file to create /opt/cisco/msc/builds//

      2. Run the below commands on node1, node2, and node3 to update packages
      a. cd /opt/cisco/msc/builds//bin/
      b. ./update_packages.sh

      for eg: ./update_packages.sh 2.3.1a

      3. Run below commands on node2 and node3 to load the image and perform the upgrade
      NOTE: this step must to be executed on node2 and node3 only before next step
      a. cd /opt/cisco/msc/builds//upgrade/
      b. ./upgrade.sh

      4. Run below command on node1 to load the image and perform the upgrade
      NOTE: you will be prompted to enter node2’s and node3’s password
      a. cd /opt/cisco/msc/builds//upgrade/
      b. ./upgrade.sh -1 -3
      for eg: ./upgrade.sh -1 1.1.1.1 -3 2.2.2.2

      – currently installed release
      – target upgrade release
      – IP address of node2
      – IP address of node3
      NOTE: node1-3 refer to the labels applied to the VMs. Labels are easily seen using
      “docker inspect –format {{.Spec.Labels}}”
      $ docker inspect msc-sjc1 –format {{.Spec.Labels}}
      map[msc-node:msc-node1]

    2. also, Mark, I just noticed that you mentioned that you did follow the readme instructions already. Sorry for the overlook. can you please let me know what sort of error you are seeing when it bombs ?

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.