Site icon

Deploying MSO on Cisco Application Service Engine (OVA based SE) — updated 1/12/2021

Soumitra Mukherji

Updated 7/06/2020 : Please skip to the SE install section: SE Release 1.1.3d (OVA) Install Instructions:

Update 1/21/2021:  Current and recommended release of SE is 1.1.3e.

Starting with Release 2.2(3) of MSO,  Cisco Application Service Engine (CASE) can be used to host the MSO.  Previous to this release MSO could only be installed in a docker-swarm environment. 

The older docker-swam based MSO is still a valid option, but the new version of MSO running on  CASE has advantages and I highly recommend that folks  migrate to that.  Migration can be done very easily by pulling in a backup and restoring the backup to the CASE based MSO.   If you are just starting off with MSO, I strongly suggest you start off with  *test out MSO running on CASE.   I suspect that in the future the CASE based MSO will become the defacto choice for the majority of customers.

* Added on 5/11/2020:

Caution:   At the moment the  SE OVA cluster recovery has not been sorted out yet.   The SE cluster requires a minimum of 2 SE nodes up to be operable.   If you have 2 SE nodes fail, then the SE cluster will be inoperable and cluster recovery process has not been defined at the moment.  I will update this blog as soon as that feature is developed and I test it out.

* Added on 5/20/2020:

SE Release 1.1.3 has just been released.  However MSO ( current release of 2.2.4e or 3.0.1i) is not qualified to work with that.  The next release of MSO — 3.0.2 (which has supposedly much richer features) will work with SE 1.1.3.  

MSO Version 2.2.4e  or 3.0.1i work fine with SE1.1.2

* Added on 7/6/2020:

So What Exactly Is CASE ?

CASE stands for Cisco Application Service Engine.  To make it easier for myself, I will just refer to it from here as Service Engine or SE.  

SE is an appliance that serves as a common platform for deploying specialized Cisco Data Center applications.  One of these applications is the newer MSO that we will talk about in this writeup.

SE runs Kubernetes (K8s) under the hood.   Because of the K8s based architecture it inherently gives you built in resiliency for your apps, lets you upgrade apps easily (recall the rolling upgrade built into K8s).   If you have worked with the docker-swarm based MSO, and have had to upgrade your MSO at any time,  you probably (like me) have struggled with upgrading MSO, trying to get the right version of python and associated modules to make it work (and probably opened a TAC case or two ultimately to get it going).   The MSO that runs on SE can be upgraded with just a few clicks in the GUI making it very convenient and easy for operations.    Also,  I have found the SE based MSO to be much more snappy/responsive.

Cisco Application Services Engine is deployed as a cluster of three service nodes. This clustering provides reliability and high-availability software framework.

7/13/2020: With the release of SE 1.1.3d, the architecture has changed.   If you are looking at 1.1.3d onwards, please directly go to the section:

SE Release 1.1.3d (OVA) Install Instructions:

and follow from there

 

Cisco Application Service Engine can be deployed in two modes :

Fabric Internal Mode:

As of this writing (April 30th, 2020), this mode is only available in the hardware base Service Engines (UCS-C series server appliances).  In this mode the hardware based Service Engine Cluster is directly connected to ACI leaves (much like APICs).   Then you have to install the ISO SE image on them and do the basic configuration on those SEs.  You finish off the install by installing the SE APP on the APIC and discovering the Service Engines.  Once this Hardware Based SE has been installed/discovered from the ACI App, you can start deploying approved Cisco Data Center management applications on that, like Network Insights Advisor and Network Insights Resources.

In this mode, inband management configuration has to be done for the ACI fabric to talk to the SE. Keep in mind that you can still keep using Out Of Band Management for all your other stuff if you were doing that before.  Even the SE mgmt interface will be on a OOB network.  The Inband management interfaces of these SEs are just for SEs to gather information from the ACI Fabric.

Figure 1

Fabric External Mode:

In this mode, you don’t need to have the Hardware SE appliance from Cisco (though you can).   In this mode, you can’t (as of yet, it may be coming later, but I really don’t know for sure), use Inband Fabric Configuration for the SE to talk to the Fabric.  This also means that NIA and NIR app in this mode will give you limited functionality.   However there are other applications that will work just wonderfully in this mode, like the MSO to start off with.

In Summary,  Fabric External Mode can be configured on:

In this write up, my intention is to take you step by step on how to install the OVA based SE on VMware and then install the appropriate MSO image on the SE.

I also added a section to this document that shows in detail (with screnshots) on how to upgrade/downgrade the SE based MSO Code using GUI or CLI.

Remember that at the end of the day, the SE is an appliance.  All the complexities of Kubernetes is hidden under the hood and as a user you cannot execute the standard K8s commands unless you go in as root.   You will not be able to go in as root unless you work for Cisco TAC / Professional Services or equivalent.  

For the curious, I will go in as root and show you some screen captures of what happens under the hood in SE cluster in the K8S level.

Let's Get Going:  

(you might want to skip this and go to the section"SE Release 1.1.3d (OVA) Install Instructions" for SE 1.1.3d install and then come back to the "Installing MSO on SE" section)

 

Before we start let’s talk about the pre-requisites.  In a production environment, you will need the following.

In a lab environment you can get away with only 1 esxi and you can install 1 SE Node instead of 3.

Assuming you already have your esxi infrastructure installed and ready to go, let’s proceed from there.  In this example, I will assume you are using vCenter to manage your esxi hosts.

Before you start, please make sure to go to vCenter and make sure that you enable NTP and sync up correct time on all your ESXi’s.  This is always good practice.

Figure 2

Download the SE image from CCO.  For this particular scenario, you need to download the OVA image. 

In vCenter right click on host and then “Deploy OVF Template”. ( You could also download the OVF file in your vCenter Content Library and install from there if you wanted to)

Figure 3

Next just follow through and go like any OVA/OVF install.  In my case I name my First SE VM,  “se01”

Figure 4

I put it on the first host

Figure 5

On the next screen just keep going by hitting the “NEXT” button

Figure 6

Choose your Storage for the SE VM

Figure 7

Choose your vswitch or dvs port group, such that you can can access the SE VMs from your local machine.  In my case, I just chose the vSwitch VM Network port group.

figure 8

Everything above is pretty standard stuff.   Now you have to fill in the OVF template, so please go a bit slow here so you don’t make mistakes.

For the first node”

Figure 9.1
Figure 9.2
Figure 9.3
Figure 9.4

Once you are done with that, the OVF will install and when it’s done, it will be in shut down state.  Boot up se01 VM now from vCenter and web console into it from vCenter.

Watch it boot.  If the boot goes some way and then goes to the screen where it takes you to the CLI Initial Setup screen, then you have probably put in the peer IP, Serial number in the wrong format.   You can try to put it in now from CLI Startup screen.  as shown below.  Hopefully you did it right the first time, so you don”t have to do it again.   In case you needed to do it, from CLI Startup screen this is what it would look like for SE01.

Figure 10

Note:  If you were installing in lab and wanted to get away with one SE Node only, then:

Once you are done with SE Node one (se01) in my case, install SE node 2 (se02) in my case. Please install in the 2nd esxi.   I will paste my entries below for se02 so you can follow through if you wanted to.

Please make sure that in the last entry for the dbgtoken,  ssh to se01 and do a “acidiag dbgtoken”   then enter that value for se02 as shown below.  When SSH’ing in, please use username of  rescue-user  and the password that you configured earlier

Figure 11.1
Figure 11.2
Figure 11.3

Repeat similar steps for se03, put in third esxi.

Figure 12.1
Figure 12.2
Figure 12.3

Finally after about 20 minutes, the K8S Cluster will be up and ready.  Check by doing “acidiag health” in any of the SEs.  Wait till the output says “All components are healthy”

Figure 12.3.1

Note: In case clustering does not want to come up, chances are that some parameter in the setup was not entered correctly.   TAC will have to go in as root and look at  “/confg/run-ovf-prepare.log” to find out which parameter was incorrect.

You are all done installing the SE Cluster !!!

Next Step,  let’s install the MSO Code.

Installing MSO on SE:

From from  Cisco App Store  or from CCO, download the MSO code.  The extension of the file will be .aci

Figure 13

Once downloaded, scp the file to any one (and one only) of the SEs.  In my case, i download to se01.

Note:   Want to make a point here.  Remember that the underlying architecture is K8S based.   The MicroServices that make up the MSO app will automatically be spread out amongst all the nodes.  This is a function of Kube-Schedular.   Please see “Distribution of MSO Apps” in figure 27.  You can also see the distribution directly from the MSO GUI, please see screenshot Figure 20.7.

The kube-api server of the master nodes will keep track of the services and initiate kube-schedular to bring them up on another node if there is a failure.   This is an inbuilt function of K8s.   For that reason, you don’t have to install the MSO app on different SEs.  Just install on any one SE node and K8S will take care of the rest.

scp “Cisco-MSO-2.2.3 (1).aci”  rescue-user@10.201.35.100:/tmp

Figure 14

Check to make sure that the code is there on that SE (se01 in my case)

Figure 15

In this particular case, the MSO code has spaces and brackets in the name.  I don’t want to deal with that.  So, I’m just renaming it to get rid of those special characters.

Figure 16

Next on the SE, do a “acidiag app -h” to see all the acidiag app commands available.

Figure 17

Install the MSO App with “acidiag app install image_name.aci”

Follow with “acidiag image show image_name”  as suggested

Figure 18

Check the status with “acidiag app show”

Follow with “acidiag app enable $ID”  where $ID is the id of the app as shown below

Figure 19

Repeat “acidiag app show” few times, till you see:

 

Figure 20.1-2
Figure 20.3

All done !  You can now point your browser to any of the SE IP addresses and the MSO GUI will come up.

Please log in with the default credentials of:

Figure 20.4

Now you could set up the MSO from scratch for a new install or you can backup your install from a previous MSO and restore it here.  After that you are ready to go.

Some Items on MSO worth Mentioning:

Figure 20.5
Figure 20.6
Figure 20.7

Upgrading/Downgrading SE based MSO Image

For the SE based MSO, upgrading / downgrading can be done:

Direct GUI Method to Upgrade/Downgrade:

Figure 20.8
Figure 20.9
Figure 20.10

A Look At the Kubernetes infrastructure for SE, for the Curious

As mentioned before, you will not have access to the K8s infrastructure as this is an appliance and all the complexities of K8s is hidden from the user.   If you could go in as root (being a Cisco employee/partner), you will be able to get access to the underlying K8s infrastructure and inspect it.

Before Installing MSO app:

Figure 21
Figure 22
Figure 23
Figure 24

After Installing MSO app:

Figure 25
Figure 26.1
Figure 26.2
Figure 26.3
Figure 27
Figure 28
Figure 29

SE Release 1.1.3d (OVA) Install Instructions:

added on 7/6/2020

Pre-requisites:

SE Release 1.1.3d is now available as of 7/3/2020.   You can install this release of SE and install MSO release 3.0.2d on there as a running K8s application.  ( Note that MSO release 3.0.2d will not install on a previous version of SE)

Below, I show you instructions with screenshots on how to do this.  Note, that in this case,  I am installiing the SE in a lab and will only use a 1 Node Master (This is O.K. for me, since this is a lab and I don’t want to use up extra disk space.  Also note that this release of SE uses less disk space compared to the previous release.  The previous release used 600 GB where as this release only uses 230 GB)

Before Installing the SE Release 1.1.3d OVA, it is essential to understand the architecture of this version of SE.

SE 1.1.3d OVA has 2 interfaces:

  1. Fabric Interface (fabric0) — this is also known as the data interface and is meant to connect to L3Out connection from ACi Fabric Leaves.  The purpose for this interface is:
    • Cisco Application Services Engine Clustering.
    • App to app communication.
    • Access the management network of the Cisco ACI fabric.
    • All app to ACI fabric comunications.
  2. Management Interface (mgmt0) – this is also known as management interface and needs to connect to OOB network. The purpose for this interface is:
    • Accessing the Cisco Application Services Engine GUI.
    • Accessing the CLI over SSH.
    • DNS and NTP.
    • Firmware uploads.
    • Intersight device connector.

The Diagram below depicts the connectivity that’s required.   Keep in mind that you cannot just use 1 interface, you have to have both interfaces connected to the proper port groups in the esxi dvs/vswitch.

Figure 29A

In my setup, to make it easier, I have just connected the 2 interfaces to different sub-interfaces of an upstream router.  This just happens to be the ISN router where ACI Spines connect to (through the vCSRs).  That way both interfaces are functional and I have both Data and Management interfaces that can be used for their respective purpose.  If I had just connected the mgmt interface to the mgmt port group,  I would be able to ssh to SE and reach any destination, but from the app that runs on the SE ( in this case MSO),  I would only have management access (ssh/web).   However, the MSO app would not be able to reach the APICs, Radius Server, etc, etc as that would require the Data Interface (in essence I would have a non functional MSO).

The connectivity I am using is depicted below.  Note that you generally would want to connect the fabric0 Interface to an L3Out on some ACI Leaf. (For NIA/NIR application running on SE that L3Out needs to have connectivity to the inband of the APIC, either through route leaking from mgmt tenant VRF or creating the L3Out for inband external on mgmt VRF.  You could also connect that L3Out to an external router that is in the same routing domain that the SE data interface (fabric0) connects to.)

Note:  You can also put both the data interface and fabric interface on the same subnet (i.e. connect it to the same dvs/vSwitch port group).   That is sometimes an easier way to do it.  Ofcourse each interface would have to have a different IP in the same subnet.  Update: 11/3/2020:   I’ve tested the same interface install in 1.1.3d and it has issues.  The install is generally not successful, and if you do manage to get it up and running, you will notice from K8s level, that the pods continiously go to crash mode and then recovers.  In other words it’s not stable.   A bug has been raised for this (CSCvv82660), however I’m not sure when or if the bug will get fixed in a SE release, given that Nexus Datacenter (the next gen SE), is around the corner and this is where everything will be going.   Incidentally, the first release of MSO that can run on ND is 3.2.   The problem occurs in single node or clustered node install for SE 1.1.3d.   Having the fabric and mgmt interfaces in different subnets works flawlessly.

Update 1/12/2021:  SE 1.1.3e works fine with both data and fabric interfaces in same subnet. 

Figure 29b

Now that we understand the connectivity requirements, installing the OVA is like any other OVA

Name the SE VM and choose the vCenter DC

Figure 30

Note that this release of SE only uses 230 GB of disk space compared to 600 GB for the previous release

Figure 31

Choose your Data Store

Figure 32

On this SE  VM there are 2 vNICs and both need to be connected. 

  1.  mgmt (management interface)
  2.  fabric0 (data interface)
Figure 33

Configure the Host Name, rescue-user password and management IP/gateway

Figure 34

Enter the Cluster Name,  the other SE (K8s)  master IPs, and this masters dbgtoken value

Figure 35

Keep populating the rest of the fields as shown in the diagram below.   Note, that since this is the 1st (and only) master in this deployment,  Idid not check the “Download config from Peers…” option

Figure 36

Verify the summary config and hit “Finish” to compete the configuration

Figure 37

After install is done, go to vMware console for the SE VM, log in as “rescue-user” and your defined password. 

Check the health of the SE (K8s) cluster by executing “acidiag health”.   In my single node cluster it took less than 15 minutes for it to be in ready state

Figure 38

You are all done installing the SE release of 1.1.3d.  Now follow from Figure 13 onwards for installing your version of MSO SE code.

Alternatively, you could also use the SE GUI for installing the MSO App as shown below.   The UI username is:  admin and password is the rescue-user password that you set during install.

Figure 39

Accessing the MSO UI directly

Once your MSO is up and running, if you wanted to access the MSO directly point your browser to  https://<mgmtIPofSE>/mso/login (if you pointed your browser to https://<mgmtIPofSE>/, then you would land up on the SE device UI)

Additional SE Cluster Health view commands

acidiag health

Figure 41

acidiag cluster get config

Figure 42

acidiag cluster get masters

Figure 43

acidiag cluster get workers

Figure 44

* Note:   in the output of “acidiag cluster get workers” command, you will notice that the output of the command does not show any workers.   The reason for that is because in this (lab environment), we have only 1 K8s (SE) Node and the Node is configured as a master.  

Question: The next question you might ask is “if that node is a master then how are we running the MSO app on that master, would the app not need a K8s worker node to run the app ? “

Answer:  The reason that you can run app on the 1 node master is that the taint has been removed from the master, so the master also acts as a worker.   This is obvious when you look at it from root.  You will notice that there are no taints configured for this 1 node K8s Master.

kubectl get nodes -o wide (after logging in as root — will need token to do that)

Figure 45

kubectl describe node dmz-se01 | grep -i taint

(notice there are no taints in the master

Figure 46

In a typical dedicated K8s Master Node configuration, the Master Node will be tainted with a “master:NoSchedule” taint, so that workloads will not be deployed on it.   Below is an example from a typical K8s Master Only Node Taint

Figure 47

acidiag app show

Fibgure 48

References:

Exit mobile version