Added on 11/24/2020: Interesting discussion on One Time Password with Ansible that customer pointed out. Please see bottom of post
update for L3Out was added on 10/8/2020: Please see the bottom of the post.
In this post, we will show how to automate Infrastructure provisioning with the MSO Controller running on Service Engine using Ansible. Towards the end of the post we also use Ansible directly with APIC to complete some site local configurations for L3Out that need to be done from APIC directly.
Please note that since we are running Ansible against MSO running on Service Engine, in the Ansible “hosts” file I define the full path to the URL dmzmso/mso/login. dmzmso is resolved in the /etc/hosts file to point to the IP if the Service Engine. If you were running MSO on docker-swarm you would just put the entry for Ansible hosts file as dmzmso and not dmzmso/mso/login. Please see Figure 13 for more details on this.
aciadmin@ubuntu-jump:~/Ansible/ansible-scripts/MSO$ cat hosts
When it comes to automation for orchestration or monitoring there are always many choices and at the end the choice depends on what you are more comfortable with. Using automation with Cisco APIC is very common and most folks are very comfortable with that. There are many choices for ACI automation, curl with bash sending json/xml encapsulated payload , direct posting from UI, postman / postman runner, python scripts, Ansible, Terraform and many others. In addition APIC has an awesome real time MIB browser that is really useful to check on objects and also to figure out where to send posts to.
Now that ACI Anywhere ( Multisite, Cloud extensions, etc, etc) is getting heavy traction with customers, orchestration by automation for MSO is a topic that’s coming up often. Just like ACI automation, MSO automation offers a variety of choices. There is also a awesome real time swagger doc built in where you can send direct queries and do configuration from through api calls.
Below figure shows an example of querying MSO objects directly from swagger docs. You can post, put, delete, patch from here directly. This is a great option for a quick and dirty configuration that you don’t have to keep repeating.
Why Ansible is a good choice for MSO automation
As you can see above, every object in MSO has a object ID. When using postman, python, etc, etc, you first need to get the object ID of the particular object and then use that in the post. This makes it a bit more involved to use postman, or python scripts to configure MSO. You first have to do a query, get the object ID and then use that in subsequent posts for that object.
With Ansible, you can just run the ansible playbooks without having to worry about all that. All the complexities have been taken out of that.
Getting Started with Ansible/MSO:
In this blog post, I will get you started on how to use Ansible with MSO. This includes:
- setting up the ansible environment
- downloading and using the updated ansible-galaxy collection for MSO
- Encrypting MSO password on the Ansible server using Ansible Vault.
The example shown here is going to be a simple ACI topology spanned across 2 sites. However the intent is that once you know how to do this you can customize this to your liking. You can also write scripts that will generate your playbooks and add entries to the variable files of Ansible for large scale repeatable orchestration.
Let’s get started:
The first thing you want to do is to setup your Ansible environment properly. You probably want to have a dedicated Ubuntu/CentOS server (VM), that is setup for doing this. You don’t want tons of folks trying to configure the fabric from their own desktop/laptop.
Note: For this example, I have used “18.04.5 LTS (Bionic Beaver)” version of Ubuntu. If you are using a different flavor/version, (for example xenial Ubuntu), the commands might be slightly different. Also, a different flavor of Ubuntu might come with a different version of python in the base OS. In that case, please adjust the commands accordingly. For this example, I would suggest using the same release, so you can just cut and paste the commands from this document.
In this example, I will show you how to setup Ansible on an Ubuntu VM. You can do the same with any linux flavor, including MAC.
When it comes time to test with the playbooks you can download the playbooks that I use in this exercise from:
To Download my playbooks:
git clone https://github.com/soumukhe/MSO-Ansible.git
I have the same link posted again right below figure 18, where you will actually build a sample tenant.
Setting up the Ansible Environment:
- You will need to have python3.x and Ansible 2.9 or higher for this to work. (please do not try to upgrade your base OS python version. That can mess up your OS. This document will guide you through installing pyenv, so you can change the working python versions at will as you want).
- Also, keep in mind that later you may have to upgrade your Ansible version for newer features and perhaps even your python version
- For this reason, it is highly recommended to setup a virtual environment for Ansible. Each Virtual Environment will have a version of Python and Ansible tied to it, so you can quickly fire up that virtual environment with a script and use the same playbooks as needed. In addition you don’t want to upgrade the base OS python version as that will break the OS. In the figure below you can see I have many releases of virtual environments with different python releases and Ansible releases available to me that I can fire up in a moment’s notice.
Below I list the instructions to do this. Keep in mind that this example is on an ubuntu VM. If you are using a different flavor of linux or even MAC, you can use equivalent commands.
Check version of python that came with system with python -V
curl -L https://raw.githubusercontent.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash
sudo apt-get update && sudo apt-get upgrade -y
sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev git
# Load pyenv automatically by adding
# the following to ~/.bashrc:
eval "$(pyenv init -)"
- log out of the ubuntu box and ssh back in for the startup variables to kick in
pyenv install -l
mkdir Ansible && cd Ansible
pyenv install 3.7.9
pyenv local 3.7.9
python3 –m pip install virtualenv
python3 –m pip install upgrade pip
mkdir envs && cd envs
python3 –m virtualenv vEnvAnsible2.9.6-P3.7.9
Create a requirmentes file ”requirements.txt” copy and paste the below:
cat > requirements.txt <<EOF
python3 –m pip install –r requirements.txt
- Copy and paste the below to make the environment startup script
cat > goAnsible2.9.6-P3.7.9 <<EOF
pyenv local 2.7.17 (you don't really need to do this, it's just to show you that the next line will bring up your correct version of python in your virtual env)
deactivate ( do not do this as you proceed with this exercise. this is just to show you how to deactivate the virtual environment)
ansible-galaxy collection install cisco.mso --force
mkdir MSO && cd MSO
Make your ansible.cfg file in the MSO folder
Make a hosts file in the MSO directory and put a name for your MSO
Add dmzmso to resolve in /etc/hosts
Using Ansible Vault to encrypt password for ansible user for MSO
echo “hello\!” | base64
or, better yet: head -c 21 /dev/urandom | base 64
ansible-vault encrypt_string aGVsbG9cIQo= --name “password” >> password.yaml
Copy the contents of the file (password.yaml) to the external_vars.yml ansible playbook variable file that we will use. Remember the contents of the file is the key of password and the value is the encrypted password. The external_vars.yml file is the variable file that the playbook will call to get the values of the variables (key/value) that we want, for e.g. vrf: sm-VRF1-ansible
Now, Execute the playbook:
# usage: ansible-playbook --ask-vault-pass vaultPlaybookTest.yaml -i hosts -vvv
# usage: ansible-playbook --vault-id vaultPass vaultPlaybookTest.yaml -i hosts -vvv
Below is an example run of the playbook. Note that when you run it it asks for the ansible vault password.
ansible-playbook --ask-vault-pass vaultPlaybookTest.yaml -i hosts -vvv
During development of the playbook, you may get tired of having to type in the password every time. You could create a file with the ansible-vault password so you don’t have to do this.
When I was encrypting the password using the “ansible-vault encrypt_string” command it had asked me for a password. I put “secret” as the password. So, now I create a file called vaultPass with that password in it.
(vEnvAnsible2.9.6-P3.7.9) aciadmin@ubuntu-jump:~/Ansible/ansible-scripts/MSO$ cat vaultPass
Now you can execute the playbook pointing to that file where your ansible vault password is (in clear text) as shown below. ( vaultPass is the file where I’ve stored the vault Password in clear text. In my case, when ansible vault asked for a password during encrypting the MSO password, I supplied the vault password a value of “secret”. )
ansible-playbook --vault-id vaultPass vaultPlaybookTest.yaml -i hosts -vvv
Below is a sample topology that I created in my environment between 2 sites. You can download the playbooks for yourself, and then modify the external_vars.yaml file based on your Fabric setup and execute them. Make sure to have the correct entry in the host inventory file (in my case it was dmzmso which resolved to the IP from /etc/hosts file).
To Download my playbooks:
git clone https://github.com/soumukhe/MSO-Ansible.git
The Ansible module documentation can be found at:
update on 10/07/2020: The previous link seems to not work any more. Please use: https://docs.ansible.com/ansible/2.9/modules/list_of_all_modules.html
Use the documentation to understand and implement your playbook. However don’t use those modules in your playbook. Use the equivalent ansible-galaxy collection modules instead. The ansible-galaxy modules have the latest updates and capabilities. Some of the normal modules will not work even though the documentation says it will.
As an example, let’s look at the figure below and concentrate on the module: mso_schema_site_anp_epg_domain
To use the galaxy-collection module, prefix the module name in the documenation with “cisco.mso.“
Please refer to Figure 12 from earlier in this writeup to see the ansible-galaxy MSO modules you installed.
An example of using the ansible-galaxy module in a playbook is shown in the figure below
What’s the next step:
- As mentioned before, Ansible is a great choice for MSO automation / orchestration
- Do not automate tasks that you will do just once, it makes no sense. Automation for repeated tasks adds value. This gives you consistency and speed
- For large repeated tasks, you may want to write a python script that will take your input values from a CSV file or excel file and create your playbooks and enter entries in your external_vars.yaml file
- Make sure that you have a quality control system for your playbooks or any automation for that matter. Remember that you can destroy your network/fabric as quick as you can make it with automation. You should have a system in place that checks for flags and errors in playbooks before executing.
I've added to the github repo ansible playbooks for L3Out. Please use git clone to get the playbooks:
git clone https://github.com/soumukhe/MSO-Ansible.git
Also, make sure to install the ansible galaxy collections for aci.
ansible-galaxy collection install cisco.aci --force
Remember that when using MSO:
The following items are created from MSO:
- association with VRF
- external EPG
- external EPG prefixes and scope
The following items are fabric-local and are created from APIC:
- Routing configuration
- L3Out Node Profile
- L3Out Interface Profile
- Other Policies for L3Out as needed
Included are some changes in the hosts file and external-vars.yaml file, a json file and a certificate directory.
Also, 3 playbooks for L3Out Ansible configuration as shown as below:
- 1ha.AddL3OutSite1Only.yaml ( this creates the L3Out/external EPG on MSO and deploys)
- 1hb.finishOffL3OutAPIC.yaml (this competes the rest of the L3Out Fabric Local configuration on APIC)
- 1h.consolidated.yaml (this calls the above 2 playbooks. You only need to run this)
Please study the json file and the playbooks. Suit it to your environment. You will also see a certificate directory with certificate file and a private key. To create the certificate and private key you can use the below openssl command:
openssl req -new -newkey rsa:2048 -days 36500 -nodes -x509 -keyout ansible.key -out ansible.crt -subj '/CN=ansible/O=cisco/C=US'
Create a local ansible user on apic and copy the contents of the CRT file to the APIC (ansible) user either from GUI or from post. (please generate your own certifcate with openssl as shown above). The post for certifcate file is included in the directory also “cert_install.yaml“.
Added 11/24/2020: Interesting Discussion that customer pointed out:
You can use certificates or ansible vault with encrypted passwords as I show in https://unofficialaciguide.com/2020/08/28/getting-started-cisco-mso-with-ansible/. If using encrypted passwords with ansible vault, the passwords would still have to be authenticated against Radius/AAA where you could see who ran the scripts and what they modified.
The issue is you are trying to use OTP (using some sort of RSA token device) and playbooks contain multiple module, it’s actually a separate run for each module and it’s a new login and it asks for the one time token each and every time for each module. Totally legitimate ask that I had not even thought of ! I don’t know how to solve that. Maybe there is a way, I just have not found it.
Comment from one of my colleagues:
It looks to me like the entire Ansible project (module) would need to be rewritten to cache the bearer token and make it available between tasks after an initial authentication with the OTP. Not the RSA token, but the bearer token. See line 252 here and Step 7 here. The issue is not an ACI issue per say. It’s Ansible with One Time Password issue.