Are you using MSO (ACI Multisite Orchestrator), in standalone mode ?   Well, please do not do that in production.  Often times in a lab environment you will be tempted to do this.  I’m guilty of doing this myself on the Cisco  ACI Advanced Bootcamp ACI training labs, simply because I’m too lazy to bring up another 2 VMs and also to save on compute resources. 

So, now what I want to discuss here is how to upgrade the standalone MSO to a newer release.  Say you upgraded APIC+Fabric from 4.1.1i to 4.2.2e.  So, you want to upgrade your Standalone MSO from 2.1.1h to 2.2.2b.   I used to do this earlier easily but for the last 2 releases doing a standalone MSO upgrade has not been successful.  I internally looked for TAC cases on this issue and noticed that there were quite a few TAC cases opened on exactly this issue and for almost all these open cases, customer ultimately just installed  brand new standalone MSO and restored the backup from the previous MSO (if they took a backup from the previous MSO, that is).  Often times in the lab situation I am guilty of not doing a backup before upgrading.

So, what’s the error message you get when the upgrade bombs ?   Unfortunately, I don’t have a live capture right now, because I had to do this for a few MSOs in a hurry because students were waiting on me.  However the error is something to the effect that it tries to do a docker upgrade and then bombs and the docker engine is shut down and the upgrade fails saying something to the effect of   “is docker running ?”

 

If this happens, go ahead and start back docker back again.

systemctl start docker

systemctl enable docker

systemctl status docker

docker   –info

 

Here is the hack to get it to work:

Optional Steps:
Do a docker upgrade manually. 

If you opted to do this optional step, then make sure that you have your proxies configured in the Docker VM (CentOS).   Assuming your Internet connectivity needs a proxy.

first ssh as root to the MSO (obviously 🙂 )

shell proxy would look something like this:

/etc/profile.d/proxy.sh:
export http_proxy=http://mylabproxy.company.com:80/
export https_proxy=http://mylabproxy.company.com:80/
export ftp_proxy=http://mylabproxy.company.com:80/
export no_proxy=.mydomain.com
export HTTP_PROXY=http://mylabproxy.company.com:80/
export HTTPS_PROXY=http://mylabproxy.company.com:80/
export FTP_PROXY=http://mylabproxy.company.com:80/

Browser Proxy would look something like this:

/etc/envoronment

export http_proxy=http://mylabproxy.company.com:80/
export https_proxy=http://mylabproxy.company.com:80/
export ftp_proxy=http://mylabproxy.company.com:80/
export no_proxy=.mydomain.com
export HTTP_PROXY=http://mylabproxy.company.com:80/
export HTTPS_PROXY=http://mylabproxy.company.com:80/
export FTP_PROXY=http://mylabproxy.company.com:80/

YUM Proy would look something like this:

/etc/yum.conf

[main]
proxy=http:mylabproxy.company.com:80/

Docker Proxy:

[Service]
Environment=”HTTP_PROXY=http://mylabproxy.company.com:80/”
Environment=”HTTPS_PROXY=http://mylabproxy.company.com:80/”

How to upgrade Docker.

yum check update
yum update -y
yum remove docker docker-common docker-selinux docker-engine
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager –add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install docker-ce
sudo systemctl start docker
sudo systemctl status docker
sudo systemctl enable docker
docker info

Required (non-optional)

1) scp the code to MSO, to directory /opt/cisco/msc/builds

e.g.

scp msc_2.2.2b.tzr.gz    root@msc_ip:/opt/cisco/msc/builds/

2) cd to directory /opt/cisco/msc/builds 

3) Untar,   tar -xvzf msc_2.2.2b.tzr.gz 

4) go to the prod-standalone directory for your release (from the untarred file)

cd /opt/cisco/msc/builds/msc_2.2.2b/prod-standalone

5)  vi the upgrade.sh file,  Search for  “upgradeDocker” and comment it out.

6) run the upgrade script —      ./upgrade.sh

The script will take some time to run ( I think 10 to 12 minutes)  and upgrade the MSO to the new release.

Check to make sure that docker swarm shows that all containers are up and replicated by using the command:  “docker service ls“

Log into MSO UI, make sure it’s working. After that go back to the SSH session. Use the command: “docker images” and you will see the old images there also.

Clean up the old images by using the following command:

“ docker system prune -a “

Below screen-shot shows, I have reclaimed 10.5GB of disk ( I had more older images also than what’s shown in above screenshot). Now, the only images remaining are for 2.2.2b