Getting Started: Cisco MSO With Ansible (updated: 10/8/2020)

update for L3Out was added on 10/8/2020:  Please see the bottom of the post.


When it comes to automation for orchestration or monitoring there are always many choices and at the end the choice depends on what you are more comfortable with.  Using automation with Cisco APIC is very common and most folks are very comfortable with that.  There are many choices for ACI automation,  curl with bash sending json/xml encapsulated payload , direct posting from UI, postman / postman runner, python scripts, Ansible, Terraform and many others.   In addition APIC has an awesome real time MIB browser that is really useful to check on objects and also to figure out where to send posts to.

Now that ACI Anywhere ( Multisite, Cloud extensions, etc, etc) is getting heavy traction with customers,  orchestration by automation for MSO is a topic that’s coming up often.   Just like ACI automation, MSO automation offers a variety of choices.  There is also a awesome real time swagger doc built in where you can send direct queries and do configuration from through api calls.

Figure 1

Below figure shows an example of querying MSO objects directly from swagger docs. You can post, put, delete, patch from here directly.  This is a great option for a quick and dirty configuration that you don’t have to keep repeating.

Figure 2

Why Ansible is a good choice for MSO automation

As you can see above,  every object in MSO has a object ID.  When using postman, python, etc, etc, you first need to get the object ID of the particular object and then use that in the post.   This makes it a bit more involved to use postman, or python scripts to configure MSO.  You first have to do a query, get the object ID and then use that in subsequent posts for that object. 

With Ansible, you can just run the ansible playbooks without having to worry about all that.  All the complexities have been taken out of that.

Getting Started with Ansible/MSO:

In this blog post, I will get you started on how to use Ansible with MSO.  This includes:

  • setting up the ansible environment
  • downloading and using the updated ansible-galaxy collection for MSO
  • Encrypting MSO password on the Ansible server using Ansible Vault.

The example shown here is going to be a simple ACI topology spanned across 2 sites.   However the intent is that once you know how to do this you can customize this to your liking.  You can also write scripts that will generate your playbooks and add entries to the variable files of Ansible for large scale repeatable orchestration.  

Let’s get started:

The first thing you want to do is to setup your Ansible environment properly.   You probably want to have a dedicated Ubuntu/CentOS server (VM), that is setup for doing this.  You don’t want tons of folks trying to configure the fabric from their own desktop/laptop.

Note:   For this example, I have used “18.04.5 LTS (Bionic Beaver)” version of Ubuntu.  If you are using a different flavor/version, (for example xenial Ubuntu), the commands might be slightly different. Also, a different flavor of Ubuntu might come with a different version of python in the base OS.  In that case, please adjust the commands accordingly.   For this example, I would suggest using the same release, so you can just cut and paste the commands from this document.

Figure 2a

In this example,  I will show you how to setup Ansible on an Ubuntu VM.  You can do the same with any linux flavor, including MAC.

When it comes time to test with the playbooks you can download the playbooks that I use in this exercise from:

To Download my playbooks:
git clone

I have the same link posted again right below figure 18, where you will actually build a sample tenant.

Setting up the Ansible Environment:

  • You will need to have python3.x and Ansible 2.9 or higher for this to work. (please do not try to upgrade your base OS python version.  That can mess up your OS.   This document will guide you through installing pyenv, so you can change the working python versions at will as you want).
  • Also, keep in mind that later you may have to upgrade your Ansible version for newer features and perhaps even your python version
  • For this reason, it is highly recommended to setup a virtual environment for Ansible.  Each Virtual Environment will have a version of Python and Ansible tied to it, so you can quickly fire up that virtual environment with a script and use the same playbooks as needed.   In addition you don’t want to upgrade the base OS python version as that will break the OS.   In the figure below you can see I have many releases of virtual environments with different python releases and Ansible releases available to me that I can fire up in a moment’s notice.
Figure 3

Installing pyenv:

Below I list the instructions to do this.  Keep in mind that this example is on an ubuntu VM.  If you are using a different flavor of linux or even MAC, you can use equivalent commands.

  • Check version of python that came with system with python -V
  • curl -L | bash
  • sudo apt-get update && sudo apt-get upgrade  -y
  • sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev git

  • # Load pyenv automatically by adding
    # the following to ~/.bashrc:
    export PATH="$HOME/.pyenv/bin:$PATH"

    eval "$(pyenv init -)"
  • log out of the ubuntu box and ssh back in for the startup variables to kick in
Figure 4
  • pyenv install -l
  • mkdir Ansible && cd Ansible
  • pyenv install 3.7.9
Figure 5
  • pyenv versions
  • pyenv local 3.7.9
Figure 6
  • python3 –m pip install virtualenv
  • python3 –m pip install upgrade pip
  • mkdir envs && cd envs
  • python3 –m virtualenv vEnvAnsible2.9.6-P3.7.9
Figure 7
  • source ./vEnvAnsible2.9.6-P3.7.9/bin/activate
  • Create a requirmentes file ”requirements.txt” copy and paste the below:

cat > requirements.txt <<EOF

Figure 8
  • python3 –m pip install –r requirements.txt
  • Copy and paste the below to make the environment startup script

cat > goAnsible2.9.6-P3.7.9 <<EOF
source ~/Ansible/envs/vEnvAnsible2.9.6-P3.7.9/bin/activate
cd ~/Ansible/ansible-scripts
return 0

Figure 9
  • mkdir ansible-scripts
  • pyenv local 2.7.17 (you don't really need to do this, it's just to show you that the next line will bring up your correct version of python in your virtual env)
  • source goAnsible2.9.6-P3.7.9
Figure 10
  • python -V
  • ansible --version
  • which python3
  • deactivate  ( do not do this as you proceed with this exercise.  this is just to show you how to deactivate the virtual environment)
Figure 11
  • ansible-galaxy collection install cisco.mso --force
  • ls /home/aciadmin/.ansible/collections/ansible_collections/cisco/mso/plugins/modules
Figure 12
  • mkdir MSO && cd MSO
  • Make your ansible.cfg file in the MSO folder
  • Make a hosts file in the MSO directory and put a name for your MSO
  • Add dmzmso to resolve in /etc/hosts
Figure 13

Using Ansible Vault to encrypt password for ansible user for MSO

  • echo “hello\!” | base64
  • or, better yet:    head -c 21 /dev/urandom | base 64 
  • ansible-vault encrypt_string aGVsbG9cIQo= --name “password” >> password.yaml
Figure 14

Copy the contents of the file (password.yaml) to the external_vars.yml  ansible playbook variable file that we will use. Remember the contents of the file is the key of password and the value is the encrypted password The external_vars.yml file is the variable file that the playbook will call to get the values of the variables (key/value) that we want, for e.g. vrf: sm-VRF1-ansible

Figure 15

Now, Execute the playbook:

# usage: ansible-playbook --ask-vault-pass vaultPlaybookTest.yaml   -i hosts -vvv
# usage: ansible-playbook --vault-id vaultPass  vaultPlaybookTest.yaml   -i hosts -vvv

Figure 16

Below is an example run of the playbook.  Note that when you run it it asks for the ansible vault password.

ansible-playbook --ask-vault-pass vaultPlaybookTest.yaml  -i hosts -vvv

Figure 17

During development of the playbook, you may get tired of having to type in the password every time.  You could create a file with the ansible-vault password so you don’t have to do this. 

Now you can execute the playbook pointing to that file where your ansible vault password is (in clear text) as shown below. ( vaultPass is the file where I’ve stored the vault Password in clear text.  In my case, when ansible vault asked for a password during encrypting the MSO password, I supplied the vault password a value of “secret”.  )

ansible-playbook --vault-id vaultPass vaultPlaybookTest.yaml -i hosts -vvv

Figure 18

Below is a sample topology that I created in my environment between 2 sites.  You can download the playbooks for yourself, and then modify the external_vars.yaml file based on your Fabric setup and execute them.  Make sure to have the correct entry in the host inventory file (in my case it was dmzmso which resolved to the IP from /etc/hosts file).

To Download my playbooks:
git clone

Figure 19

Important Note:

The Ansible module documentation can be found at:

update on 10/07/2020:  The previous link seems to not work any more.  Please use:

Use the documentation to understand and implement your playbook.  However don’t use those modules in your playbook.  Use the equivalent ansible-galaxy collection modules instead.    The ansible-galaxy modules have the latest updates and capabilities.  Some of the normal modules will not work even though the documentation says it will.

As an example, let’s look at the figure below and concentrate on the module: mso_schema_site_anp_epg_domain 

Figure 20

To use the galaxy-collection module, prefix the module name in the documenation with “cisco.mso.

Please refer to Figure 12  from earlier in this writeup to see the ansible-galaxy MSO modules you installed.

An example of using the ansible-galaxy module in a playbook is shown in the figure below

Figure 21

What’s the next step:

  • As mentioned before, Ansible is a great choice for MSO automation / orchestration
  • Do not automate tasks that you will do just once, it makes no sense.  Automation for repeated tasks adds value.  This gives you consistency and speed
  • For large repeated tasks, you may want to write a python script that will take your input values from a CSV file or excel file and create your playbooks and enter entries in your external_vars.yaml file
  • Make sure that you have a quality control system for your playbooks or any automation for that matter.  Remember that you can destroy your network/fabric as quick as you can make it with automation.  You should have a system in place that checks for flags and errors in playbooks before executing.

Update 10/8/2020:

I've added to the github repo ansible playbooks for  L3Out. Please use git clone to get the playbooks:

git clone

Also, make sure to install the ansible galaxy collections for aci.

ansible-galaxy collection install cisco.aci --force

Remember that when using MSO:

The following items are created from MSO:

  • L3Out
  • association with VRF
  • external EPG
  • external EPG prefixes and scope

The following items are fabric-local and are created from APIC:

  • Routing configuration
  • L3Out Node Profile
  • L3Out Interface Profile
  • Other Policies for L3Out as needed

Included are some changes in the hosts file and external-vars.yaml file, a json file and a certificate directory.

Also,  3 playbooks for L3Out Ansible configuration as shown as below:

Figure 22

Please study the json file and the playbooks.   Suit it to your environment.  You will also see a certificate directory with certificate file and a private key. To create the certificate and private key you can use the below openssl command:

openssl req -new -newkey rsa:2048 -days 36500 -nodes -x509 -keyout ansible.key -out ansible.crt -subj '/CN=ansible/O=cisco/C=US'

Create a local ansible user on apic and copy the contents of the CRT file to the APIC (ansible) user either from GUI or from post. (please generate your own certifcate with openssl as shown above). The post for certifcate file is included in the directory also “cert_install.yaml“.

4 thoughts on “Getting Started: Cisco MSO With Ansible (updated: 10/8/2020)

  1. Devarshi, now that Terraform has support for MSO, could you provide guidance on how to get setup and running? From my perspective, Terraform is a superior tool for automating ACI and likely MSO. Also, how do you see customers that are primarily network-centric handling daily operations? For example, since MSO will not do any of the “fabric” work like creation of VPCs or interface selector updates and most of the daily work is associated with that and static bindings for EPGs, do you see customers leaving those bindings out of the MSO templates for EPGs?

    1. Hi Dan, I’ll let Devarshi answer. However, there are few things I want to point out.
      * I personally have not worked much with Terraform other than some very basic dabbling around. I really like the concept of Terraform and I know that Cisco has a team working on Terraform/ACI/MSO support modules. I am really busy right now, trying to get some other stuff done, and my intention is to dive into Terraform soon as possible, do some testing and write some guidance article out as soon as I get comfortable with it.
      * Though I think Terraform might be really worth exploring and that might be the platform of choice, I generally don’t tell customers what to do directly or compare products directly when it comes to 3rd party products (other than Cisco (since I do work for Cisco and I like our products :)). The reason is that some customers become very religious and they have preferences and sometimes they get upset if you tell them about some other product that might be more efficient. So, my tactic is to show customers all solutions and let them make the decision – while stating my personal opinion. This generally turns out to a positive outcome because the customers realize that you are looking out for them.
      * On the comment on MSO not doing the site local fabric configurations, like access-policies, etc, etc, generally customers who have a lot of bare metals do that with some sort of ACI level automation like Ansible / postman. However, customers who are fully integrated with VMM domains (vmWare/K8s), don’t need to do access policy configurations unless they are bringing in newer compute nodes. So, for fully ACI Integrated customers this is a 1 time day 1 configuration only and they don’t need to bother about it generally. The same holds for cloud sites. I think this is where Terraform will really shine because you could do a full end to end configutation with it.
      * As far as customers using automation with MSO, from what I’ve seen (atleast the customers that I’ve been involved with), they are not doing automation yet, but are really interested about it. That is because with all this ACI Anywhere, especially cloud support Multisite/MSO is now getting to be a more common deployment method.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.