Changes:
11/30/2021 with updated information on Ansible and Terraform
In a previous article, I had introduced and guided on how to install Cisco Nexus Dashboard. In this post I will discuss and do a short guidance on installing Cisco MSO on Cisco ND.
Installation is really simple as you will see in a bit. It’s really not much more involved than going to ND Service Catalog / APP Store and then download and install the MSO Application. Before we go there let’s quickly discuss some of the main items that you should know about MSO on ND.
Items you should know about MSO on ND:
- MSO version 3.2.x or above will be able to run on ND. Currently ND release is version 2.0.x and runs only on hardware platform only. For running MSO on ND, vND (Virtual ND on vmWare), will be out soon and you should be able to also run MSO on vND. Soon you should be able to also run ND on Cloud (AWS) at which time you could run MSO on ND on Cloud.
- MSO version 3.2.x is the first version for MSO on ND and currently this version does not have cloud site (AWS/Azure) support. Recall from the ND article that Sites are on-boarded from ND and not from MSO. ND release 2.0 does not allow Cloud Site onboarding. The next release of MSO (3.3) and ND (2.1) will have cloud support.
- Implications: If you have cloud ACI Sites or hybrid ACI Fabric, please wait for the next release of MSO and ND to come out. If you have only physical fabrics, you should not have any issues.
- As mentioned in the ND article, ND and Apps running on ND are tightly integrated.
- Implications:
- Sites are onboarded on ND. On MSO the sites that you onboarded from ND will show up. All you have to do is Enable the Sites and configure the Infra. If you were going to backup from an older release of MSO and intended to restore to the MSO/ND release, it is imperative that you name the sites in ND just like the Sites were named in the older MSO, otherwise the restore on the ND/MSO will fail complaining about unknown sites.
- User Creation (local or remote ( radius/tacacs)) are not an option in MSO any more. The users local or remote are the ones that are defined from ND. Remote Users can do cross launch (SSO) for APIC controllers without need to enter username password again.
- API: Because of the tight integration of ND and it’s apps, some of the APIs that were previously available directly on MSO are not available from MSO any more, i.e. depricated. However orchestration and getting information from MSO using API is fully supported. In other words, you can use API calls with json encapsulation using Postman, bash with curl, Python, etc, etc to do what you need to do. Later in this article, I will show an example of a simple Bash script with curl to demonstrate this.
- Ansible: Ansible in the current release 3.2 does not work. This is being addressed and will soon be available. Ofcourse you could modify the underlying Ansible Python code and make it work, but this will be a 1 off. I would suggest waiting till this is fixed in the publicly released version of MSO Ansible modules.
11/30/2021: Ansible modules can be used with NDO now with httpapi connections - Terraform: Terraform in MSO has not worked till now unless you are using Terraform MSO Raw modules (i.e. send Json encapsulated payload). However in my opinion this defeats the purpose of using Terraform since Terraform State is not kept. This is being addressed and will soon get fixed. I will update the post when this starts working in Terraform with Native MSO Modules.
11/30/2021: Terraform modules / resources work now with NDO. (resources are present for building objects from NDO for physical sites and AWS, Azure and GCP are still missing)
- Implications:
- Upgrading MSO from previous releases. This has not changed. You can backup from an older release and restore the backup on this release. Please read article Upgrading ACI Fabrics and MSO. If you were going to backup from an older release of MSO and intended to restore to the MSO/ND release, it is imperative that you name the sites in ND just like the Sites were named in the older MSO, otherwise the restore on the MSO/ND will fail complaining about unknown sites.
Short guidance on installing Cisco MSO on Cisco ND.
As discussed earlier, installing any app on ND is really simple. If you wanted to download the MSO software on a http server or locally, and then install from there you can Click on Service Catalog and click on Install Apps

At that point, you can choose Remote or Local and install the App from there as shown below

Alternatively ( the preferred method), is to click on the APP Store tab and choose the MSO App from there and install it from there as shown below

Click on Agree and Download

You will see the state change from Uploading to Installing to Installed.

At this point, click on Enable.

Next, you are going to be presented with a “Deployment Profile”. The Deployment Profile is nothing but a K8s CRD (Custom Resource Definition). At this point there are 2 different CRDs that are defined and you can use any of those. In the future you will be able to define your own CRDs depending on the App and your needs.

It will take a little bit of time for the app to deploy and get ready for use. When it’s ready, you can open up the MSO app by clicking on the “Open” button.

Once you open it, you will be greeted with the familiar Welcome Banner

From here, you can either configure it as a new MSO, by changing the Sites to Managed (remember the Sites came directly from ND). and then configure the Infra.


Or, if you are upgrading from an older MSO to this release, restore the backup from the older MSO to this MSO.
- If you were going to backup from an older release of MSO and intended to restore to the MSO/ND release, it is imperative that you name the sites in ND just like the Sites were named in the older MSO, otherwise the restore on the MSO/ND will fail complaining about unknown sites.
- All the Sites that were in the backup of the previous release need to be in the site list for this release of MSO running on ND (otherwise backup will fail)
- You will have to unmanage the site first before restore and then manage it back after restore


As discussed earlier, please notice that from 3.2 MSO does not have “Add Site” option and there is no “Add User” Option.

Example Bash Script to send Json encapsulated API calls to MSO
As discussed above API calls to MSO/ND work just like they used to in previous release of MSO. Below is a sample bash script that you can use to send json encapsulated API calls to MSO
#!/bin/sh
# Make sure to have jq installed. ubuntu: "sudo apt update && sudo apt install jq -y "
# MSO 3.1.1g / SE
# USER=someUser # only needed for Option 1
# Password=somePassword # only needed for Option 1
# HOST=10.0.0.3
# MSO 3.2.1f / ND
# USER=someUser # only needed for Option 1
# Password=somePassword # only needed for Option 1
HOST=10.0.0.5
# POST
CURL_OPTS_POST='-v -k -X POST'
# GET
CURL_OPTS_GET='-s -k -X GET'
### Option 1. Login & Get Token, credentials come from variables
#AUTHPAYLOAD={\"username\":\"$USER\",\"password\":\"$PASS\"}
#TOKEN=$(curl $CURL_OPTS_POST -H "Content-Type: application/json" https://$HOST/api/v1/auth/login -d $AUTHPAYLOAD | jq -r .token)
#echo $TOKEN
### Option 2. Login & Get Token, credentials come from file like example creds.json
## make a creds.json file like so: {"username" : "someUser" , "password": "somePassword"}
TOKEN=$(curl $CURL_OPTS_POST -H "Content-Type: application/json" https://$HOST/api/v1/auth/login -d "@creds.json" | jq -r .token)
#echo $TOKEN
# Get Tenants
echo "Tenant Info"
echo "----------"
curl $CURL_OPTS_GET -H "Content-Type: application/json" https://$HOST/api/v1/tenants -H "Authorization: Bearer ${TOKEN}" | jq
# Get Schemas and extract Contract Relationships
echo " Contract Relationship"
echo "----------------------"
curl $CURL_OPTS_GET -H "Content-Type: application/json" https://$HOST/api/v1/schemas -H "Authorization: Bearer ${TOKEN}" | jq '.schemas[].templates[].anps[].epgs[] | {name, contractRelationships}'
The Output gives me the Contract Relationships as you can see below:

References:
Cisco Multi-Site Configuration Guide for ACI Fabrics, Release 3.2(x)
Please see Inline for answers:
——————————–
Thanks for the two parts ND write-up, really informative!! I have some questions regards ND with MSO hope you could help.
[SM] Thanks very much for the appreciation !
1. I want to deploy ND nodes to different locations, two in one DC and one in another DC. So how should I initialize the cluster? I am planning to initialize them in the same location and the ship them out to destination. But can the CIMC IP addresses be changed after cluster is built?
[SM]You can do either way:
a) bring up cluster after you put the ND nodes in the correct location (2 in 1 site and 1 in another site)
b) do it the way you are proposing. The CIMC IP is only used during bootstrap to obtain the Serial # of the node and to allow configuration for bootstrap using serial over CMC. Once node is booted it’s not needed. You can change the CIMC IP in the tradional way from UI of CIMC or command line. That has no bearing on anything any more.
I do want to point out something that I recently learnt. From ND release 2.02 there is a requirement that the Data(fabric) Interfaces of all the nodes need to be in the same L2 network. So, if you had different sites, you would need to stretch that L2 network somehow across the sites. You could use ACI Stretched EPG/BD across sites (with BUM) enabled for this purpose. This requirement is so that /32 host IP information is needed for streaming telemetry (other possibilities are being explored like bgp with /32 advertisement). So, please keep that in mind.
Update: I just learnt: The above L2 stretch requirement is only needed for DCNM, not for ACI. (as mentioned, a solution for that will be out soon). Regardless if your focus is ACI, this is not something you should be concerned about.
2. Regarding that ND node data network needs access to ACI inband MGMT, will they also need access to ACI OOB MGMT as well? If not, how will the MSO on ND talk to APICs?
[SM]MSO on ND uses the OOB mgmt network of ND to talk to APIC OOB. You will need APIC OOB for this purpose.
3. Will there be any obvious downside if I used the same subnet/network for ND node data and management networks?
[SM] I don’t think there is any downside, unless you run into some bug. I bring this up because SE1.1.3.d had a problem with that that they fixed in SE 1.1.3e. However I want to let you know that Cisco BU is not fond of that. Every time I bring that up they don’t like it. For all practicality, I would suggest that if you do have hardware NDs, you should use different subnets. For the VM based ND (coming out soon), I often see that customers don’t want to go through the hassle of having 2 separate subnets. Keep in mind that if you deployed the VM (or cloud based), you would probably only target one app like MSO to run on that ND (though some limited combinations are supported). If you want the full suite of applications, you should use the hardware based ND. After you pay the VMware tax and everything having the VM based ND does not save you any money.
Thanks,
/S
Please see inline:
——————-
Thanks for the info…I am planning to use 10.10.0.0/24, 10.10.10.0/24 and 10.10.20.0/24 for site 1 ND CIMC, data and management networks respectively; 10.20.0.0/24, 10.20.10.0/24 and 10.20.20.0/24 for site 2 ND CIMC, data and management networks respectively. Regarding initialization, if I put the nodes in correct locations with correct CIMC addresses, I would have to make 10.10.0.0/24 and 10.20.0.0/24 communicate with each other between sites in order to build the cluster, right?
[SM] Yes, that’s correct. The OOB networks should be able to route to each other ideally.
If so, wouldnt it be easier if I just bring up three node CIMCs on 10.10.0.0/24 to initialize the cluster while putting in the real data and management IP addresses for each node. Then ship the 3rd node to site 2 and reconfigure CIMC to be on 10.20.0.0/24?
[SM] Bringing up the cluster would be faster the way you are doing it. But either way it would work. The good thing about doing it the other way is that your end state of CIMC has already been configured from before, so you don’t have to touch it again.
Do you know what is the reasoning behind this new requirement for v2.0.2? So if I deployed ND V2.0.1 today without the L2 reachability for the Data network for nodes, what would be the impact if I needed to do the upgrade to v2.0.2?
[SM] I just updated the previous thread. I did some more research and found out that that requirement is not needed for ACI. Only for DCNM. So you should be good. Also, Cisco will soon come up with a better solution for that.
Thanks! Appreciate the quick reply!