Part 2 - Installing AppDynamics APM Agents to report application data to AppD Controller -

Introduction

With AppDynamics you can configure a machine agent and application agents to stream telemetry to your AppDynamics Controller dashboard to provide information to aid in monitoring and troubleshooting specific issues with the application running within your Cisco ACI deployment. This guide is written to setup a machine agent, network visibility agent, DB agent and guide you through getting them to send data to the AppDynamics controller. This article is not a full guide on setting up multiple Agents monitoring application exit calls, which is need for data to appear in Cisco Nexus Dashboard Insights application. You need to monitor application exit calls for Network Data to be captured from the AppDynamics agents. Another important caveat to know is that ONLY JAVA and .NET Application’s report Network Data into AppDynamics. Meaning you will only see data in NDI if you are monitoring a JAVA or .NET application.

In a later article we will cover utilizing Cisco ACI integrated Kubernetes and AppDynamics Application Agent Auto-Instrumentation feature to make this agent setup much easier as well as showing the different metrics captured by AppD that are fed into Nexus Dashboard Insights.

Integration Requirements

You will need 1 VM to run the AppD stack and 1 VM to be your host application server, two VMs in total. Reachability is needed between both VMs in order to send the data between the nodes. I have covered setting up the AppD controller in a previous article and will not be covering that here.

For our application server I am using a Ubuntu 20.04 VM with the following specs. Feel free to copy them so you wont run into any issues.

Screen Shot 2021-11-29 at 8.17.14 PM — Figure 1: Screenshot of my Application server VM Hardware specs

AppD Architecture

AppD is broken into 4 different pieces. You have your AppD enterprise console which is used to manage all your AppD Controllers. Then you have your AppD controller which is used to create the necessary agents that will communicate with the AppD controller. The agents generated from the controller will only talk to that controller. Running alongside the controller on the same servevr is your Events Service which keeps track of all the data and anomalies that are captured from the agents. Finally, we have the AppD agents (Machine, Application, DB, Analytics, Cluster, Network) which run with our Apps or Servers. These agents report back to the controller about the performance of the app you are monitoring.

AppD Machineagent Installation

Figure 3: Screenshot of AppD Controller MachineAgent Wizard

Choose the Variables that are correct for your environment. Since we are monitoring a 64bit Ubuntu 20.04 Linux distro the choices above will work. Hit the download button and scp that file to the host you want to monitor. Create a directory for Agent and move the zip file into that new directory.

cisco@php-test:~$ mv machineagent-bundle-64bit-linux-21.7.0.3160.zip machineagent
cisco@php-test:~$ cd machineagent/
cisco@php-test:~/machineagent$ ls
machineagent-bundle-64bit-linux-21.7.0.3160.zip
cisco@php-test:~/machineagent$ unzip machineagent-bundle-64bit-linux-21.7.0.3160.zip

From your machineagent directory we now have access to the agent. Let’s do the following to install the agent into the server.

cisco@php-test:~/machineagent$ pwd
/home/cisco/machineagent/jre/bin/


jre/bin/java -jar machineagent.jar &> /dev/null #You can use this to hide the output and run the JAVA app in the background.


cisco@php-test:~/machineagent$ jre/bin/java -jar machineagent.jar
Using Java Version [11.0.11] for Agent
Using Agent Version [Machine Agent v21.7.0-3160 GA compatible with 4.4.1.0 Build Date 2021-07-23 23:04:28]
[INFO] Agent logging directory set to: [/home/cisco/machineagent/logs]
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/home/cisco/machineagent/lib/guice-4.2.3.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Machine Agent Install Directory :/home/cisco/machineagent
Machine Agent Temp Directory :/home/cisco/machineagent/tmp
Tasks Root Directory :/home/cisco/machineagent/controlchannel
Redirecting all logging statements to the configured logger
03:06:39.205 [system-thread-0] DEBUG com.appdynamics.common.framework.util.EmbeddedModeSecurityManager - Installed
03:06:39.215 [system-thread-0] INFO com.appdynamics.analytics.agent.AnalyticsAgent - Starting analytics agent with arguments [-p, /home/cisco/machineagent/monitors/analytics-agent/conf/analytics-agent.properties, -yr, analytics-agent.yml]

Started AppDynamics Machine Agent Successfully.

Then hit the continue button at the bottom of the page. You should now see your Server’s host name in the GUI.

Figure 4: Screenshot showing the machine agent is reporting correctly.

Enabling Server Visibility

https://docs.appdynamics.com/4.5.x/en/infrastructure-visibility/server-visibility/enable-server-visibility

Now that the machine agent is installed, we have to enable server Visibility. Open a second SSH session to the php-test VM and do the following.

cd to machineagent directory and edit the controller-info file to enable server monitoring

https://docs.appdynamics.com/4.5.x/en/infrastructure-visibility/machine-agent/configure-the-machine-agent/where-to-specify-machine-agent-configuration

cisco@php-test:~$ cd machineagent 
cisco@php-test:~/machineagent$ nano conf/controller-info.xml

Figure 5: Screenshot showing the line you can search on and then will change to enable Server Visibility Monitoring.

Kill the running agent however you see fit (pgrep java then kill -9 <pid>) and restart the agent to load the new configuration by running the JAVA command we ran previously. Navigate back to your AppD controller GUI and reload the page. You should now see some new data filtering in. It will take some time for it to populate.

cisco@php-test:~/machineagent$ jre/bin/java -jar machineagent.jar

Figure 6: Screenshot showing a healthy and enable Server Visibility reporting.

Figure 7: Screenshot showing HW utlization reporting into AppD Controller.

Figure 8: Screenshot showing the all the metrics being reported.

After some time we have let our machine run and we have found that there are some issues. AppD made us aware of the physical issues on the box.

Figure 9: Screenshot showing an error with our Application server being reported by the AppD MachineAgent’s.

Figure 10: Screenshot showing the specific errors after double-clicking on our application server in the AppD Controller GUI.

You can see that there is something that is eating into our HDD , and we are running out of SWAP partition space.

Installing the Network Visibility Agent

https://docs.appdynamics.com/21.3/en/infrastructure-visibility/network-visibility/network-visibility-supported-environments

https://docs.appdynamics.com/21.3/en/infrastructure-visibility/network-visibility/set-up-network-visibility-on-linux/install-the-network-agent-on-linux

First, we must ensure our host has the proper packages installed in the system.

You must install these required libraries before installing the Network Agent host. You can use the following for Ubuntu 16.04

sudo apt-get install -y libdl.so.2
sudo apt-get install -y libpthread.so.0
sudo apt-get install -y librt.so.1
sudo apt-get install -y libm.so.6
sudo apt-get install -y libc.so.6
sudo apt-get install -y libncurses.so.5
sudo apt-get install -y libtinfo.so.5
sudo apt-get install -y libstdc++.so.6
sudo apt-get install -y libgcc_s.so.1

Now download and install the agent from AppD’s website

Figure 11: Screenshot showing AppD network visibility installation documents.

Figure 12: Screenshot of the Network Visibility Agent in the AppD software download site.

Download using curl directly to the machine or download locally and transfer the Network Agent to your VM. Be sure to filter appropriately and download the version for your AppD controller version.

Now follow AppDs instruction on how to install their .deb file.

sudo dpkg -i appd-netviz-x64-linux-21.3.0.2181.deb


cisco@php-test:~$ sudo dpkg -i appd-netviz-x64-linux-21.3.0.2181.deb
Selecting previously unselected package appd-netviz.
(Reading database ... 188781 files and directories currently installed.)
Preparing to unpack appd-netviz-x64-linux-21.3.0.2181.deb ...
Unpacking appd-netviz (21.3.0.2181) ...
Setting up appd-netviz (21.3.0.2181) ...
Created symlink /etc/systemd/system/multi-user.target.wants/appd-netviz.service → /lib/systemd/system/appd-netviz.service.
Processing triggers for systemd (245.4-4ubuntu3.11) ...
cisco@php-test:

Check that the service has started and is in a running status.

sudo service appd-netviz status

Figure 13: Screenshot showing a successfully running Network Visibility service.

If it hasn’t started go ahead and try to start it by doing the following.

service appd-netviz start

You will have to populate some data into the AppD, I did this by repeatedly transferring a file into the server. This created a network load I was able to view from within the AppD GUI. Not the graph below. There are 2 colored lines, perhaps it doesn’t come through in the screenshot, but the 2 boxes inside of the larger red box surrounding the graph. There is a purple line that signifies change and a blue line that is the recorded baseline for that metric. You will see this in many of AppD graphs.

Figure 14: Screenshot showing the network data from our application server.

Configuring an NodeJS and MongoDB application

First let’s create an application environment on the your Application server. We will first need to install NodeJS.

sudo apt install nodejs
sudo apt install npm

Next we will install mongodb and configure it to be used on our host.

sudo apt install mongodb

We can now edit our mongo.conf

sudo nano /etc/mongodb.conf

# mongodb.conf
# Where to store the data.
dbpath=/var/lib/mongodb

#where to log
logpath=/var/log/mongodb/mongodb.log
logappend=true
bind_ip = 0.0.0.0 #Change to 0.0.0.0
#port = 27017



systemctl daemon-reload
systemctl restart mongodb

Now let’s download our application from git.

https://github.com/DanWahlin/NodeExpressMongoDBDockerApp

git clone https://github.com/DanWahlin/NodeExpressMongoDBDockerApp

Now we can CD into the directory and begin configuring mongoDB to interact with our NodeJS application. Edit your file look like mine below. TAKE NOTE OF THE DATABASE NAME. We will need to create a database with the same name in the next step.

cd  NodeExpressMongoDBDockerApp
cd config
cat config.development.json

{
    "databaseConfig": {
        "host": "0.0.0.0",
        "database": "funWithDocker"
    }
}

Now let’s create that database. First drop into the mongo CLI.

mongo

Then we will create a database for our application with the same name as the .json above.

https://www.tutorialspoint.com/mongodb/mongodb_create_database.htm

use funWithDocker

This will create a DB. Now we need to prepare our nodejs environment. First cd back into our main app directory. Then install the necessary modules. The NPM command is node.js package installer and if you are behind a proxy you will need to npm to interact with your proxy environment.

cd  NodeExpressMongoDBDockerApp
npm install

Now we need to create a NodeJS Appdynamics Agent.

Installing the DB Agent

Installing the DB agent is very straightforward as well. Simply navigate to your AppDynamics Controller and go to your Getting Started Wizard and select Databases. Go ahead and select MongoDB.

Figure 15: Screenshot of the AppD Controller GUI Database agent wizard.

Next, we will download the DB agent from our controller and SCP the file over to our application server. I moved the dbagent zip into its own directory. Unzipped the file. Set our JAVA_HOME environment variable and finally start the app.

THE DB AGENT IS NOT PACKAGED WITH JAVA. YOU WILL MAKE USE OF THE MACHINEAGENT JAVA DEPLOYMENT.

Figure 16: Screenshot showing a successfully connected AppD DB Agent connection. There is still one more step which is to create configure a user. Note you are Step 5 in the AppD Wizard

scp db-agent-21.7.0.2434.zip user@<IP OF APPLICATION SERVER>:~
mdkir dbagent
mv db-agent-21.7.0.2434.zip dbagent
cd dbagent
unzip dbagent
export JAVA_HOME=/home/cisco/machineagent/jre/
./start-dbagent

The agent will now connect to the AppD Controller, but we are not finished setting up the Agent. We still need to create a user with the correct permissions for AppD. Let’s configure our Database collector.

Open another CLI session to your application server and drop into the mongo CLI. Then enter the following command block, you can edit this user however you see fit. The information will need to match what we do in the next step.

mongo

use admin
db.createUser({ user: "tanujaAdmin",
pwd: "tanuja123",
    roles: [
     { role: "clusterMonitor", db: "admin" },
     { role: "read", db: "admin" },
    ]
})

Now take this username and password and enter it into the AppDynamics GUI.

Figure 17: Screenshot of adding the newly created DB user to the AppD Agent configuration.

After some time you will see data in the AppD Controller.

Create a Linux systemd service’s to run the agent’s in the background

NOTE: THIS IS AN IMPORTANT STEP IF YOU WANT YOUR AGENTS TO BE PERSISTENT ON REBOOT

To run the machine agent in the background on my Linux system, I created a systemd service to monitor and manage the agent. You can consider using it as well.

cd /usr/lib/systemd/system

cisco@php-test:/usr/lib/systemd/system$ sudo cat  appd-machineagent.service

[Unit]
Description=Machine Agent Damon

[Service]
ExecStart=/home/cisco/machineagent/jre/bin/java -jar /home/cisco/machineagent/machineagent.jar
User=cisco

[Install]
WantedBy=multi-user.target

cisco@php-test:/usr/lib/systemd/system$

cisco@php-test:/usr/lib/systemd/system$ sudo chmod +x appd-machineagent.service

cisco@php-test:/usr/lib/systemd/system$ systemctl start appd-machineagent.service
cisco@php-test:/usr/lib/systemd/system$ systemctl enable appd-machineagent.service
cisco@php-test:/usr/lib/systemd/system$ systemctl status appd-machineagent.service

NOTE: I have created system services for the following. We will configure doing many of these in later steps. You can run the apps in your terminal session, in the background, or by creating a service. Pick your poison.

cisco@php-test:/etc/systemd/system$ cat appd-machineagent.service
[Unit]
Description=Machineagent Service aleccham

[Service]
ExecStart=/home/cisco/machineagent/jre/bin/java -jar /home/cisco/machineagent/machineagent.jar
User=cisco

[Install]
WantedBy=multi-user.target


cisco@php-test:/etc/systemd/system$ cat appd-dbagent.service
[Unit]
Description=DB Agent Service Aleccham

[Service]
ExecStart=/home/cisco/machineagent/jre/bin/java -jar /home/cisco/dbagent/db-agent.jar
User=cisco

[Install]
WantedBy=multi-user.target



cisco@php-test:/etc/systemd/system$ cat nodejs.service
[Unit]
Description=nodejs app

[Service]
ExecStart=/usr/bin/node /home/cisco/NodeExpressMongoDBDockerApp/server.js
User=cisco

[Install]
WantedBy=multi-user.target
cisco@php-test:/etc/systemd/system$