Cisco ACI APIC and cAPIC monitoring using Prometheus/Grafana -

Changes: 1/14/2022: Replaced all occurances of APIC 5.3 to 5.2.3

From Release 5.2.3 of APIC and release 25.x of cAPIC Promethus Node Exporter Feature is available. This allows you to monitor statistics from Prometheus and view pre-configured dashboards from Grafana to get quick status of APICs/cAPICs.

In this write up, I will show you how to set up and bring up Prometheus/Grafana to monitor and visualize stats so you can try this out for yourself for POC

What is Prometheus/Grafana:

Prometheus is an Open Source monitoring solution
Started at SoundCloud around 2012-2013 and was made public in early 2015
It was inspired by Google’s Borgman, which uses time-series data as a datasource, to send alerts based on the data
Prometheus provides Metrics & Alerting. It uses Dimensional Data, time series are identified by metric name and a set of key/value pairs
Prometheus stores metrics in memory and local disk in an own custom, efficient format
Prometheus is written in GO
Prometheus includs a Flexible Query Language
Visualizations can be shown using a built-in-expression browser or with Grafana integration.
There are tons of pre-built Grafana dashboards that you can download easily by the dashboard number from https://grafana.com/grafana/dashboards
It fits very well in the cloud native infrastructure
Many client Libraries and Integrations are available
Prometheus collects metrics from monitored targets by scraping metrics (http endpoints)
A single Prometheus server is able to ingest up to 1 million samples per second as serveral million time series

Let’s get started on this, so you can try this out for yourself for a POC

First, let’s go ahead and enable Prometheus Node Exporter in cAPIC 25.x (this will be supported in APIC 5.2.3 and the procedure will be exactly the same)

on APIC/cAPIC go to System Configuration / Management Access and click on the edit button for https as shown in the figure below

Figure 1: Enabling Prometheus Node Exporter on APIC/cAPIC step1

On the next page do the below:

Enable Node Exporter
use port tcp 443
Allow Origins: https://127.0.0.1:8000
Confirm that the web service on APIC/cAPIC will reboot (no disruption to anything, other than tempoary disruption on your connection to APIC/cAPIC GUI)

Figure 2: Enabling Promethes on APIC/cAPIC step 2

Repeat the steps for all APIC 5.2.3 / cAPIC 25.x that you wish to monitor

Now let’s go ahead and install docker containers for bringing up Prometheus/Grafana

1) You need to have a VM that's already running docker and docker-compose. In case you don't have this and need help, please see the bottom part or the readme file at: https://github.com/soumukhe/apic-prometheus#readme
2) git clone https://github.com/soumukhe/apic-prometheus.git
3) cd apic-prometheus/prometheus
4) vi prometheus.yml and modify line 65, 69, 74, 78 as appropriate
5) Add more APICs/cAPICs as appropriate as indicated in line 82

Figure 3: Modifying the prometheus.yml file (step 4 & 5 above)

6) docker-compose up -d
7) execute the command: docker ps --format '{{ .Names }}' | sort |nl
(make sure you have the 5 below containers running)
    1 apic-prometheus_alertmanager_1
    2 apic-prometheus_cadvisor_1
    3 apic-prometheus_grafana_1
    4 apic-prometheus_node-exporter_1
    5 apic-prometheus_prometheus_1

Prometheus is all setup and ready to go. Now browse to the IP of the Ubuntu VM on port 9090 using http. In my case I browse to http://10.1.100.13:9090.

The Prometheus interface will come up. In the Expressions type in “node_memory_Active_anon_bytes” and you can see the statistics for that in Table format or Graph format.

Notice that as you type in the Expressions field, you will be presented with all the different statistics parameters that you can look at. Please look at different statistics to see which ones can be important for you.

Fkgure 3: Looking at Prometheus Graph for node_memory_Active_anon_bytes

Now, let’s go to Grafana and create a dashboard:

Point your browser to  http://<ip_of_ubuntu_vm>:3000.  
In my case since my vm ip is 10.1.100.13,  
I browsed to http://10.1.100.13:3000.   
The username to login is admin and the password is cisco.   
The password value is configured in grafana/config.monitoring file