Added on 12/03/2020:  A temporary workaround if you are using One Time Password / RSA device with Ansible/MSO.   Please see bottom of post.

Added on 11/24/2020: Interesting discussion on One Time Password with Ansible that customer pointed out.  Please see bottom of post

update for L3Out was added on 10/8/2020:  Please see the bottom of the post.

updated ansible rest-api module related information 1/13/2021.  Please see the bottom of the post.

In this post, we will show how to automate Infrastructure provisioning with the MSO Controller running on Service Engine using Ansible.   Towards the end of the post we also use Ansible directly with APIC to complete some site local configurations for L3Out that need to be done from APIC directly.  

Please note that since we are running Ansible against MSO running on Service Engine,  in the Ansible  “hosts” file I define the full path to the URL dmzmso/mso/login.    dmzmso  is resolved in the /etc/hosts file to point to the IP if the Service Engine.  If you were running MSO on docker-swarm you would just put the entry for Ansible hosts file as dmzmso and not dmzmso/mso/login.    Please see Figure 13 for more details on this.

aciadmin@ubuntu-jump:~/Ansible/ansible-scripts/MSO$ cat hosts
[mso]
dmzmso/mso/login

Introduction:

When it comes to automation for orchestration or monitoring there are always many choices and at the end the choice depends on what you are more comfortable with.  Using automation with Cisco APIC is very common and most folks are very comfortable with that.  There are many choices for ACI automation,  curl with bash sending json/xml encapsulated payload , direct posting from UI, postman / postman runner, python scripts, Ansible, Terraform and many others.   In addition APIC has an awesome real time MIB browser that is really useful to check on objects and also to figure out where to send posts to.

Now that ACI Anywhere ( Multisite, Cloud extensions, etc, etc) is getting heavy traction with customers,  orchestration by automation for MSO is a topic that’s coming up often.   Just like ACI automation, MSO automation offers a variety of choices.  There is also a awesome real time swagger doc built in where you can send direct queries and do configuration from through api calls.

Below figure shows an example of querying MSO objects directly from swagger docs. You can post, put, delete, patch from here directly.  This is a great option for a quick and dirty configuration that you don’t have to keep repeating.

Why Ansible is a good choice for MSO automation

As you can see above,  every object in MSO has a object ID.  When using postman, python, etc, etc, you first need to get the object ID of the particular object and then use that in the post.   This makes it a bit more involved to use postman, or python scripts to configure MSO.  You first have to do a query, get the object ID and then use that in subsequent posts for that object. 

With Ansible, you can just run the ansible playbooks without having to worry about all that.  All the complexities have been taken out of that.

Getting Started with Ansible/MSO:

In this blog post, I will get you started on how to use Ansible with MSO.  This includes:

• setting up the ansible environment
• downloading and using the updated ansible-galaxy collection for MSO
• Encrypting MSO password on the Ansible server using Ansible Vault.

The example shown here is going to be a simple ACI topology spanned across 2 sites.   However the intent is that once you know how to do this you can customize this to your liking.  You can also write scripts that will generate your playbooks and add entries to the variable files of Ansible for large scale repeatable orchestration.  

Let’s get started:

The first thing you want to do is to setup your Ansible environment properly.   You probably want to have a dedicated Ubuntu/CentOS server (VM), that is setup for doing this.  You don’t want tons of folks trying to configure the fabric from their own desktop/laptop.

Note:   For this example, I have used “18.04.5 LTS (Bionic Beaver)” version of Ubuntu.  If you are using a different flavor/version, (for example xenial Ubuntu), the commands might be slightly different. Also, a different flavor of Ubuntu might come with a different version of python in the base OS.  In that case, please adjust the commands accordingly.   For this example, I would suggest using the same release, so you can just cut and paste the commands from this document.

In this example,  I will show you how to setup Ansible on an Ubuntu VM.  You can do the same with any linux flavor, including MAC.

When it comes time to test with the playbooks you can download the playbooks that I use in this exercise from:

To Download my playbooks:
git clone https://github.com/soumukhe/MSO-Ansible.git

I have the same link posted again right below figure 18, where you will actually build a sample tenant.

Setting up the Ansible Environment:

• You will need to have python3.x and Ansible 2.9 or higher for this to work. (please do not try to upgrade your base OS python version.  That can mess up your OS.   This document will guide you through installing pyenv, so you can change the working python versions at will as you want).
• Also, keep in mind that later you may have to upgrade your Ansible version for newer features and perhaps even your python version
• For this reason, it is highly recommended to setup a virtual environment for Ansible.  Each Virtual Environment will have a version of Python and Ansible tied to it, so you can quickly fire up that virtual environment with a script and use the same playbooks as needed.   In addition you don’t want to upgrade the base OS python version as that will break the OS.   In the figure below you can see I have many releases of virtual environments with different python releases and Ansible releases available to me that I can fire up in a moment’s notice.

Installing pyenv:

Below I list the instructions to do this.  Keep in mind that this example is on an ubuntu VM.  If you are using a different flavor of linux or even MAC, you can use equivalent commands.

• Check version of python that came with system with python -V
• curl -L https://raw.githubusercontent.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash
• sudo apt-get update && sudo apt-get upgrade  -y
• sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev git
  
  
• # Load pyenv automatically by adding
# the following to ~/.bashrc:
export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
• log out of the ubuntu box and ssh back in for the startup variables to kick in

• pyenv install -l
• mkdir Ansible && cd Ansible
• pyenv install 3.7.9

• pyenv versions
• pyenv local 3.7.9

• python3 –m pip install virtualenv
• python3 –m pip install upgrade pip
• mkdir envs && cd envs
• python3 –m virtualenv vEnvAnsible2.9.6-P3.7.9

Note:   In case you wanted to use pip3 and install virtualenv, then do it this way:
sudo apt-get install python3-pip
sudo pip3 install virtualenv
virtualenv vEnvAnsible2.9.6-P3.7.9

• source ./vEnvAnsible2.9.6-P3.7.9/bin/activate
• Create a requirmentes file ”requirements.txt” copy and paste the below:

cat > requirements.txt <<EOF
ansible==2.9.6
certifi==2019.11.28
cffi==1.14.0
chardet==3.0.4
cryptography==2.8
pyopenssl
idna==2.9
Jinja2==2.11.1
MarkupSafe==1.1.1
pycparser==2.20
PyYAML==5.3.1
requests==2.23.0
six==1.14.0
urllib3==1.25.8
EOF

• python3 –m pip install –r requirements.txt
• Copy and paste the below to make the environment startup script

cat > goAnsible2.9.6-P3.7.9 <<EOF
#!/bin/sh
source ~/Ansible/envs/vEnvAnsible2.9.6-P3.7.9/bin/activate
cd ~/Ansible/ansible-scripts
return 0
EOF

• mkdir ansible-scripts
• pyenv local 2.7.17 (you don't really need to do this, it's just to show you that the next line will bring up your correct version of python in your virtual env)
• source goAnsible2.9.6-P3.7.9

• python -V
• ansible --version
• which python3
• deactivate  ( do not do this as you proceed with this exercise.  this is just to show you how to deactivate the virtual environment)


• ansible-galaxy collection install cisco.mso --force
• ls /home/aciadmin/.ansible/collections/ansible_collections/cisco/mso/plugins/modules

• mkdir MSO && cd MSO
• Make your ansible.cfg file in the MSO folder
• Make a hosts file in the MSO directory and put a name for your MSO
• Add dmzmso to resolve in /etc/hosts

Using Ansible Vault to encrypt password for ansible user for MSO

• echo “hello\!” | base64
• or, better yet:    head -c 21 /dev/urandom | base 64 
• ansible-vault encrypt_string aGVsbG9cIQo= --name “password” >> password.yaml

Copy the contents of the file (password.yaml) to the external_vars.yml  ansible playbook variable file that we will use. Remember the contents of the file is the key of password and the value is the encrypted password.  The external_vars.yml file is the variable file that the playbook will call to get the values of the variables (key/value) that we want, for e.g. vrf: sm-VRF1-ansible

Now, Execute the playbook:

# usage: ansible-playbook --ask-vault-pass vaultPlaybookTest.yaml   -i hosts -vvv
# usage: ansible-playbook --vault-id vaultPass  vaultPlaybookTest.yaml   -i hosts -vvv

Below is an example run of the playbook.  Note that when you run it it asks for the ansible vault password.

ansible-playbook --ask-vault-pass vaultPlaybookTest.yaml  -i hosts -vvv

During development of the playbook, you may get tired of having to type in the password every time.  You could create a file with the ansible-vault password so you don’t have to do this. 

When I was encrypting the password using the “ansible-vault encrypt_string” command it had asked me for a password.  I put “secret” as the password.  So, now I create a file called vaultPass with that password in it.

(vEnvAnsible2.9.6-P3.7.9) aciadmin@ubuntu-jump:~/Ansible/ansible-scripts/MSO$ cat vaultPass
secret

Now you can execute the playbook pointing to that file where your ansible vault password is (in clear text) as shown below. ( vaultPass is the file where I’ve stored the vault Password in clear text.  In my case, when ansible vault asked for a password during encrypting the MSO password, I supplied the vault password a value of “secret”.  )

ansible-playbook --vault-id vaultPass vaultPlaybookTest.yaml -i hosts -vvv

Below is a sample topology that I created in my environment between 2 sites.  You can download the playbooks for yourself, and then modify the external_vars.yaml file based on your Fabric setup and execute them.  Make sure to have the correct entry in the host inventory file (in my case it was dmzmso which resolved to the IP from /etc/hosts file).

To Download my playbooks:
git clone https://github.com/soumukhe/MSO-Ansible.git

Important Note:

The Ansible module documentation can be found at: https://docs.ansible.com/ansible/latest/modules/list_of_all_modules.html

update on 10/07/2020:  The previous link seems to not work any more.  Please use:  https://docs.ansible.com/ansible/2.9/modules/list_of_all_modules.html

Use the documentation to understand and implement your playbook.  However don’t use those modules in your playbook.  Use the equivalent ansible-galaxy collection modules instead.    The ansible-galaxy modules have the latest updates and capabilities.  Some of the normal modules will not work even though the documentation says it will.

As an example, let’s look at the figure below and concentrate on the module: mso_schema_site_anp_epg_domain 

To use the galaxy-collection module, prefix the module name in the documenation with “cisco.mso.“

Please refer to Figure 12  from earlier in this writeup to see the ansible-galaxy MSO modules you installed.

An example of using the ansible-galaxy module in a playbook is shown in the figure below

What’s the next step:

• As mentioned before, Ansible is a great choice for MSO automation / orchestration
• Do not automate tasks that you will do just once, it makes no sense.  Automation for repeated tasks adds value.  This gives you consistency and speed
• For large repeated tasks, you may want to write a python script that will take your input values from a CSV file or excel file and create your playbooks and enter entries in your external_vars.yaml file
• Make sure that you have a quality control system for your playbooks or any automation for that matter.  Remember that you can destroy your network/fabric as quick as you can make it with automation.  You should have a system in place that checks for flags and errors in playbooks before executing.

Update 10/8/2020:

I've added to the github repo ansible playbooks for  L3Out. Please use git clone to get the playbooks:

git clone https://github.com/soumukhe/MSO-Ansible.git


Also, make sure to install the ansible galaxy collections for aci.

ansible-galaxy collection install cisco.aci --force

Remember that when using MSO:

The following items are created from MSO:

• L3Out
• association with VRF
• external EPG
• external EPG prefixes and scope

The following items are fabric-local and are created from APIC:

• Routing configuration
• L3Out Node Profile
• L3Out Interface Profile
• Other Policies for L3Out as needed

Included are some changes in the hosts file and external-vars.yaml file, a json file and a certificate directory.

Also,  3 playbooks for L3Out Ansible configuration as shown as below:

• 1ha.AddL3OutSite1Only.yaml ( this creates the L3Out/external EPG on MSO and deploys)
• 1hb.finishOffL3OutAPIC.yaml (this competes the rest of the L3Out Fabric Local configuration on APIC)
• 1h.consolidated.yaml (this calls the above 2 playbooks.  You only need to run this)

Please study the json file and the playbooks.   Suit it to your environment.  You will also see a certificate directory with certificate file and a private key. To create the certificate and private key you can use the below openssl command:

openssl req -new -newkey rsa:2048 -days 36500 -nodes -x509 -keyout ansible.key -out ansible.crt -subj '/CN=ansible/O=cisco/C=US'

Create a local ansible user on apic and copy the contents of the CRT file to the APIC (ansible) user either from GUI or from post. (please generate your own certifcate with openssl as shown above). The post for certifcate file is included in the directory also “cert_install.yaml“.

Added 11/24/2020: Interesting Discussion that customer pointed out: 

You can use certificates or ansible vault with encrypted passwords as I show in https://unofficialaciguide.com/2020/08/28/getting-started-cisco-mso-with-ansible/.   If using encrypted passwords with ansible vault, the passwords would still have to be authenticated ( unless you are using local users) against Radius/AAA where you could see who ran the scripts and what they modified.

The issue is you are trying to use OTP (using some sort of RSA token device)  and playbooks contain multiple module, it’s actually a separate run for each module and it’s a new login and it asks for the one time token each and every time for each module.  Totally legitimate ask that I had not even thought of !  I don’t know how to solve that.  Maybe there is a way,  I just have not found it.

Comment from one of my colleagues:

It looks to me like the entire Ansible project (module) would need to be rewritten to cache the bearer token and make it available between tasks after an initial authentication with the OTP.  Not the RSA token, but the bearer token.  See line 252 here and Step 7 here.    The issue is not an ACI issue per say. It’s Ansible with One Time Password issue.

Added: 12/03/2020: Temporary workaround for OTP/RSA device with Ansible/MSO:

please see: https://github.com/soumukhe/MSO-OTP-RSA.git

Also Note:

In the case of OTP the password has to be put in based on what you get from your RSA Token device.   Thus you cannot use a locally stored static password in ansible vault and use that.

For that reason you have to configure the variables such that it asks for the password during run.   You also should mask the password value that you enter when running the playbook (so, it’s not visible on the screen when you type it in).   Please see the sample playbook in the workaround GIT location, to see how to do that.

Diagram attached below:

Information on Ansible MSO Rest Module.

To Install the Ansible MSO Rest Module, please see the README file.
Quick Instructions:
Make sure you starup ansible virutual env (as shown in this writup earlier)  source goAnsible2.9.6-P3.7.9
mkdir temp
cd temp
git clone https://github.com/CiscoDevNet/ansible-mso.git
cd ansible-mso
git pull origin master
ansible-galaxy collection build --force
ansible-galaxy collection install cisco-mso-* --force

Usage example for Ansible Rest Module:
I've added the 3 playbooks in the GIT repo.

1. 1ed.restQuery.yaml
2. 1ef.deleteContractUsingRestPut-withExtJson.yaml
3. 1ef.deleteContractUsingRestPut-withExtJson.yaml

1. If I wanted to do a query,  I can use the playbook 1ed.restQuery.yaml
   This gives me the json output for my intended Schema.  To Get the schema ID that’s used in the above playbook,  I can do a Ansible MSO query on everything, or I can go to the MSO Swagger query and query for all Schema Lists “mso/api/v1/schemas”  
2. Now, I want to modify the configuration with a put call.  So, I went to MSO Swagger and ran the “get schema with id” call.  I put in my schema ID, that happens to be 5fff49395100001c1701af94.  Then I downloaded the json file for the schema.  Next, I modified the json file and deleted contract associations, in effect making it like this: “contractRelationships”: []
   
3. Next, I run the playbook 1ef.deleteContractUsingRestPut-withExtJson.yaml, which basically does a  rest-api put call to MSO ( using  1ef.deleteContractUsingRestPut-withExtJson.yaml which sends the modified json file  to MSO) and then runs the deploy playbook.    The contents of 1ef.deleteContractUsingRestPut-withExtJson.yaml, is shown below:

---
- import_playbook: 1ef.deleteContractUsingRestPut.yaml 
- import_playbook: 1g.DeploySchema.yaml

#- import_playbook: 1ef.addContractUsingRestPut.yaml

#usage: ansible-playbook --vault-id vaultPass 1eg.cosolidatedContractRest.save.yaml -i hosts -vvv
#Note:
#Please run this playbook instead of the components
#Please make sure to add the cert to APIC for usr ansible (for APIC only)
#Please make sure to have the json file for L3Out in this directory also
#