Drilling down to a single server overview shows request velocity, cpu load, busy/idle workers and even bytes sent over time.

Going to throughtput shows throughtput details over time.

Workers section shows you what is happening with the workers. If you have a lot of busy workers you can see in what state their are and that might provide some insight to what is going on.

Installing Apache HTTPd agent for New Relic

1) I’m using the MeetMe New Relic Agent to monitor Apache HTTPd. It’s written in python and we’ll need to install pip. The following is using the Ubuntu python-pip package. You can find alternative install methods from pip docs.

apt-get install python-pip libyaml-dev python-dev

2) Next use pip to install newrelic-plugin-agent. When I ran it I got some errors but it still worked.

pip install newrelic-plugin-agent

3) Next we’ll need to create the configuration file for the agent. You can start by copying /opt/newrelic-plugin-agent/newrelic-plugin-agent.cfg or just use what I have posted below. The first thing you need is to set license_key. You can find your license key from your account settings page on rpm.newrelic.com. The second one is to add apache_httpd configuration. You can add multiple httpds to monitor.

cat - <<EOF>> /etc/newrelic/newrelic-plugin-agent.cfg
%YAML 1.2
---
Application:
 license_key: YOUR_LICENSE_KEY
 wake_interval: 60
 #newrelic_api_timeout: 10
 #proxy: http://localhost:8080

 apache_httpd:
  - name: localhost
    scheme: http
    host: localhost
    verify_ssl_cert: true
    port: 80
    path: /server-status

Daemon:
 user: newrelic
 pidfile: /var/run/newrelic/newrelic-plugin-agent.pid

Logging:
 formatters:
   verbose:
     format: '%(levelname) -10s %(asctime)s %(process)-6d %(processName) -15s %(threadName)-10s %(name) -45s %(funcName) -25s L%(lineno)-6d: %(message)s'
 handlers:
   file:
     class : logging.handlers.RotatingFileHandler
     formatter: verbose
     filename: /var/log/newrelic/newrelic-plugin-agent.log
     maxBytes: 10485760
     backupCount: 3
 loggers:
   newrelic_plugin_agent:
     level: INFO
     propagate: True
     handlers: [console, file]
   requests:
     level: ERROR
     propagate: True
     handlers: [console, file]
EOF

4) Make sure you have enabled mod_status in your apache and you’ve allowed access from the host your agent is running if it’s not running in the same host as your apache.

5) Then we need to add a init script for the newrelic-plugin-agent. There’s one under /opt/newrelic-plugin-agent but for me it was incomplete file so I just downloaded the one from github.

wget https://raw.githubusercontent.com/MeetMe/newrelic-plugin-agent/master/etc/init.d/newrelic-plugin-agent.deb
mv newrelic-plugin-agent.deb /etc/init.d/newrelic-plugin-agent
chmod 755 /etc/init.d/newrelic-plugin-agent
update-rc.d newrelic-plugin-agent defaults

Now you can start the newrelic-plugin-agent with

service newrelic-plugin-agent start

Now in few minutes you should see your Apache HTTPd server(s) listed under Plugins HTTPd on rpm.newrelic.com.

From the overview you can see the number of active and idle connections as well as the request rate.

From connections you’ll even more connection details. With very little connections and requests my graphs are currently slightly boring. In addition to connection details you can find more details about requests, upstreams, servers and cache.

Installing New Relic Monitoring Agent for Nginx

1) First you need to add the Ubuntu package repository for Nginx. If you’ve done this already when you installed Nginx you can skip to next step. If you are not using Ubuntu 14.04 like I am you can find the other Linux packages from Nginx website.

wget http://nginx.org/keys/nginx_signing.key
apt-key add nginx_signing.key

cat - <<-EOF >> /etc/apt/sources.list.d/nginx.list
deb http://nginx.org/packages/ubuntu/ trusty nginx
deb-src http://nginx.org/packages/ubuntu/ trusty nginx
EOF

apt-get update

2) Next you need to install the Nginx New Relic Agent

apt-get install nginx-nr-agent

3) Next you’ll need to edit the agent configuration file in /etc/nginx-nr-agent/nginx-nr-agent.ini. You need to add your license key which you can find from your account settings page on rpm.newrelic.com.

newrelic_license_key=YOUR_LICENSE_KEY

Additionally you need to add a new source which points to your Nginx status url.

[source1]
name=localhost
url=http://localhost/nginx_stub_status

4) You’ll need to add a server block to Nginx for the status. Since I had very simple configuration in my Nginx I just added the following to /etc/nginx/sites-enabled/default

server {
   listen 127.0.0.1:80;
   server_name localhost;

   location = /nginx_stub_status {
     stub_status on;
     allow 127.0.0.1;
     deny all;
   }
}

5) Last thing you need to do is reload Nginx and start the Nginx New Relic Agent.

service nginx reload
service nginx-nr-agent start

Now in few minutes you should start seeing your Nginx server listed under Plugins Nginx on rpm.newrelic.com.

New Relic has a wonderful plugin framework and there’s a ton of ready made plugins and also SDK and API for things it doesn’t already support. MySQL plugin is one of those ready made plugins and it provides all the key information you’ll need. The MySQL plugin page quickly shows what’s going on all monitored databases.

When you drill down to a individual database server the overview shows the SQL volume and how it’s split between reads and writes. More key metrics are displayed under Key Utilizations. You’ll also find database connections and network traffic on this page.

Going further down to Query analysis you’ll see in more details about the queries.

If you are using InnoDB there’s a separate page to show key metrics from InnoDB.

Installing MySQL / MariaDB Monitoring

1) MySQL plugin can easily be installed with New Relic platform installer. So the first thing you need to do is install the platform installer. You’ll need your New Relic license key which you can find from account settings on rpm.newrelic.com. Once you have that you can install it with following one liner which is for 64bit Debian and Ubuntu.

LICENSE_KEY=YOUR_LICENSE_KEY bash -c "$(curl -sSL https://download.newrelic.com/npi/release/install-npi-linux-debian-x64.sh)"

2) Next go to the newly created newrelic-npi directory and run install. You’ll want to answer yes to all the questions and when prompted to configure the plugin grab the configuration from the next step.

./npi install nrmysql

3) If you skipped configuration you can configure the plugin afterward too. You can find the configuration file under newrelic-npi from plugins/com.newrelic.plugins.mysql.instance/newrelic_mysql_plugin-2.0.0/config/plugin.json. Below is a sample configuration for MariaDB (works for MySQL) running on localhost and we’ll be creating a separate user newrelic with password somepassword which the plugin will use to gather data. You can connect to multiple databases with the same agent. I usually install this agent on the same server my nagios is running on.

{
 "agents": [
   {
     "name" : "MariaDB on localhost",
     "host" : "localhost",
     "metrics" : "status,newrelic,buffer_pool_stats,innodb_status,innodb_mutex",
     "user" : "newrelic",
     "passwd" : "somepassword"
   }
 ]
}

4) Now we need to create a user in the database and grant some rights to it.

cat - <<EOF | mysql -u root -p
CREATE USER newrelic@'%' IDENTIFIED BY 'somepassword';
GRANT PROCESS,REPLICATION CLIENT ON *.* TO newrelic@'%';
EOF

5) Last thing is to start the service but before we do make sure you have Java installed as this agent is written in Java. If you don’t have Java installed check my unattended Java install script. Otherwise you can proceed to start the service that should have been created during npi install if you answered all the questions correctly.

service newrelic_plugin_com.newrelic.plugins.mysql.instance start

Now it may take few minutes before you see your server under Plugins MySQL in rpm.newrelic.com. If it doesn’t check the log under plugins/com.newrelic.plugins.mysql.instance/newrelic_mysql_plugin-2.0.0/logs/ for hints and make sure the agent actually started.

The servers listing gives a nice overview of all servers and you an easily see if there’s any issues.

When looking at a specific server you’ll see a history of it’s CPU and memory usage as well as load average and network I/O. If you have any APM enabled applications installed you’ll see a overview of their response times, throughput and error rate. You’ll also see some of the top processes running on the server.

When you drill down to processes listing you’ll quickly see the top memory and cpu consumers. You can also look at the history of individual processes.

Installing New Relic Server Monitoring on a Ubuntu / Debian Server

1) Add an apt source for New Relic.

cat - <<-EOF >> /etc/apt/sources.list.d/newrelic.list 
# newrelic repository list 
deb http://apt.newrelic.com/debian/ newrelic non-free
EOF

2) You’ll need to get the key for New Relic repository and then update apt sources. After that you can install newrelic-sysmond.

apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 0xB31B29E5548C16BF
apt-get update
apt-get install newrelic-sysmond

3) Next you’ll need to tell it your license key so that it reports the data to your account. You can find your license key from your account settings page on rpm.newrelic.com. You can either edit the configuration file or you can set the license like shown below:

/usr/sbin/nrsysmond-config --set license_key=YOUR_LICENSE_KEY

4) Finally once everything is configured you can start the system monitor daemon.

service newrelic-sysmond start

Now in few minutes you should start seeing your server listed under Servers on rpm.newrelic.com

New Relic has seven parts or products as they call them. Those are APM, Insights, Mobile, Browser, Synthetics, Servers and Plugins. I have myself used only APM, Browser, Server and Plugins which are included in the free Lite edition.

APM

APM is the application monitoring part. It focuses on providing information about the application itself. The Lite edition shows you response times, throughput, web transaction information. It’s basically a low impact profiler. With the Pro subscription you get much deeper analysis of time spent on executing SQL, JVM statistics etc.

Browser

The browser provides insights on client side performance. Even though your application might respond quickly the users perceived performance could be poor because of network performance or even how the page is rendering on the browser.

Servers

Servers as the name suggests provides performance information about the actual server your applications are running on.

Plugins

There’s a ton of plugins to provide monitoring capabilities to systems not otherwise supported by New Relic and with it’s SDKs and API you can build your own plugins. Some of the plugins I have used are for MySQL/MariaDB, Nginx and Apache.

Mobile

Mobile is APM for mobile applications.

Synthetics

Synthetics allows you to test your application from around the world. It can check business critical user flows and interaction to make sure your site available and functioning from around the world.

Insights

Insights is a paid feature that combines business metrics with performance data. It can combine data from APM, Browser, Mobile and Synthetics for deeper analysis and segmentation and filtering.

Since I recently installed bunch of new servers and I had to refresh my memory on how I installed and configured each of the agents so I decided to write a series of articles on each of them. Here’s a list of topics I’m going to publish and as I publish them I’ll link the topic to the article. These topics will cover APM with Java and PHP, Servers and Plugins for MySQL, Nginx and Apache

• Monitoring Ubuntu / Debian Servers
• Monitoring MariaDB / MySQL
• Monitoring Nginx
• Monitoring Apache
• Monitoring Tomcat
• Monitoring PHP Applications

Now you might wonder why would you need MRTG if you already have Nagios. Nagios operates on the present value although there’s a add-on nagios grapher that can create graphs like MRTG does but I like MRTG more because you can see all the graphs on one page. Being able to see a full overview of the system is very important when trying to identify performance problems. This is also why you want to get more information out of the application, Liferay in this case. My script will help in reading connection pool and thread pool utilization as well as heap usage. Those are essential when doing troubleshooting.

First you’ll need to install and setup MRTG. I’m not going to go into details on that because it dependes on your system and the internet is full of guides to do it. Once you have it done you’ll need to download my mrtg-jmx4perl.pl script which is available in my github repository. For rest of this post I’m going to assume it’s located in /usr/local/bin/mrtg-jmx4perl.pl but it’s up to you where you put it. Just adjust the script path accordingly.

Monitoring c3p0 connection pool

Getting the values for c3p0 is a little bit tricky because it will generate a unique mbean name based on the identity token it generates for the connection pool every time the server is started. Because of this my script assumes you only have one c3p0 connection pool if you have multiple pool you’ll need to add additional logic in to the mrtg-jmx4perl to find the correct mbean. Notice that this is the case if you configure Liferay to use connection pool from portal-ext.properties instead of using a JNDI resource. We can read the mbean for c3p0 by using mbean name as “com.mchange.v2.c3p0:type=*,*” and the attributes we are most interested are numConnectionsAllUsers and numBusyConnectionsAllUsers. Below is a sample mrtg configuration snippet.

Target[dbpool]: `/usr/local/bin/mrtg-jmx4perl.pl --server=servername --mbean="com.mchange.v2.c3p0:type=*,*" --attribute="numConnectionsAllUsers numBusyConnectionsAllUsers"`
MaxBytes[dbpool]: 20
Title[dbpool]: DB Pool
PageTop[dbpool]: <h1>DB Pool</h1>
WithPeak[dbpool]: dwmy
Unscaled[dbpool]: dwmy
Options[dbpool]: growright,unknaszero,nopercent,gauge
YLegend[dbpool]: Connections
ShortLegend[dbpool]: 
LegendI[dbpool]: Connections
LegendO[dbpool]: Busy Connections
Legend1[dbpool]: Connections
Legend2[dbpool]: Busy Connections
Legend3[dbpool]: Peak Connections
Legend4[dbpool]: Peak Busy Connections

Here’s a daily graph from one of my Liferay portal servers.

Monitoring Tomcat AJP Thread Pool

This one is pretty easy because the the mbean name is static but it does vary depending on do tomcat version and connector you are using. In Tomcat 7 with native library the name for ajp thread pool is Catalina:type=ThreadPool,name=”ajp-apr-8009″. Without native library it would be ajp-bio-8009. In tomcat 6 my ajp pool mbean name is Catalina:type=ThreadPool,name=jk-8009. Notice the lack of double quotes in the name. You can easily check the name using jconsole. So for this one the config looks like:

Target[ajp-threadpool]: `/usr/local/bin/mrtg-jmx4perl.pl --server=servername --mbean="Catalina:type=ThreadPool,name=\"ajp-apr-8009\"" --attribute="currentThreadCount currentThreadsBusy"`
MaxBytes[ajp-threadpool]: 50
Title[ajp-threadpool]: AJP Thread Pool
PageTop[ajp-threadpool]: <h1>AJP Thread Pool</h1>
WithPeak[ajp-threadpool]: dwmy
#Unscaled[ajp-threadpool]: dwmy
Options[ajp-threadpool]:  growright,unknaszero,nopercent,gauge
YLegend[ajp-threadpool]: Threads
ShortLegend[ajp-threadpool]: 
LegendI[ajp-threadpool]: Threads
LegendO[ajp-threadpool]: Busy Threads
Legend1[ajp-threadpool]: Threads
Legend2[ajp-threadpool]: Busy Threads
Legend3[ajp-threadpool]: Peak Threads
Legend4[ajp-threadpool]: Peak Busy Threads

Here’s a daily graph of a thread pool.

Monitoring Heap Usage

The last one we are going to monitor is Java Heap usage. It can be read from java.lang:type=Memory using attribute HeapMemoryUsage and path used. Now we are reading only one value.

Target[heap]: `/usr/local/bin/mrtg-jmx4perl.pl --server=servername --mbean="java.lang:type=Memory" --attribute="HeapMemoryUsage" --path="used"`
MaxBytes[heap]: 1296302080
Title[heap]: Heap
PageTop[heap]: <h1>Heap</h1>
WithPeak[heap]: dwmy
Unscaled[heap]: dwmy
Options[heap]:  growright,unknaszero,nopercent,gauge,noo
YLegend[heap]: bytes
ShortLegend[heap]: 
kilo[heap]: 1024
LegendI[heap]: Used
Legend1[heap]: Used
Legend3[heap]: Peak Used

Here’s a daily graph of heap memory usage.

You can download the full sample-mrtg.cfg from github.

That’s how easy it is to monitor and graph Liferay or pretty much any Java webapp using MRTG. You could easily use this to monitor ehcache utilization or anything else that’s accessible via JMX.

There’s a nice agent called Jolokia that can provide a http bridge to JMX. You can install it as java agent in pretty much any java app or deploy it as a webapp. With Jolokia installed you can query any MBeans for their values using a simple http GET and get the data as JSON objects. JMX4Perl is a perl module and scripts that provide a easy way to run those queries through Jolokia. One of those scripts is check_jmx4perl which can be used in nagios service checks.

Okay so now we know that we are going to need Nagios, Jolokia and JMX4Perl to monitor the Liferay JVM but what should we monitor? Well that depends on what information you are interested in but at minimum I would monitor ajp or http thread usage as well as heap utilization. Just by monitoring those values you’ll know when your JVM is becomes unresponsive and can also get some early warning that there’s issues for example heap usage goes over warning threashold and never returns to normal or keeps constantly going over the threshold which could indicate they you don’t have enough heap allocated.

I’m going to assume that you have  nagios installed and configured and I will only go through how to install Jolokia and configure some checks for threads and heap. So let’s start by installing JMX4Perl.

Installing JMX4Perl is pretty simple with cpan. You just launch cpan command line client and install it like this:

cpan> install JMX::Jmx4Perl

Next you’ll need to download Jolokia and deploy the jolokia.war to your app server. For this example I’m going to assume that you are using Tomcat 7. Once you’ve deployed Jolokia it’s usually good idea to restrict who can query it. For this example we are just going to restrict it to a certain IP address (the Nagios server) and limit it to read operations only. Since I don’t like modifying the war we are going to tell Jolokia where to find the policy file through a context parameter. Create a jolokia.xml in tomcat/conf/Catalina/localhost with following content:

<Context path="/jolokia">
        <Parameter name="policyLocation" value="file:///etc/jolokia/jolokia-access.xml" />
</Context>

That tells Jolokia to look for the policy file jolokia-access.xml from /etc/jolokia/jolokia-access.xml. This is great when you are running multiple tomcats in the same server and want them to share the jolokia policy file.

Now go ahead and create the jolokia-access.xml in /etc/jolokia

<?xml version="1.0" encoding="utf-8"?>
<restrict>
        <remote>
                <host>[YOUR NAGIOS SERVER IP]</host>
        </remote>
        <http>
                <method>get</method>
                <method>post</method>
        </http>
        <commands>
                <command>read</command>
        </commands>
</restrict>

Next we need to create configuration for jmx4perl. In /etc/jmx4perl/jmx4perl.cfg we are going to include some preconfigured checks extend them. Tomcat 7 you need to add quotes around the thread pool name. We also need to set warning and critical levels for alerts. You’ll also need to add a Server for each tomcat you want to monitor.

# Default definitions
include default/memory.cfg
include default/tomcat.cfg

# ==========================
# Check definitions

<Check tc7_connector_threads>
	Use = relative_base($1,$2)
	Label = Connector $0 : $BASE
	Value = Catalina:name="$0",type=ThreadPool/currentThreadCount
	Base = Catalina:name="$0",type=ThreadPool/maxThreads
	Critical 95
	Warning 90
</Check>

<Check j4p_memory_heap>
	Use memory_heap
	Critical 95
	Warning 90
</Check>

<Server tomcat>
	Url http://MY_TOMCAT_HOSTNAME:8080/jolokia
</Server>

Then in /etc/nagios3/commands.cfg we’ll need to add a check command for jmx4perl and we’ll use the check_jmx4perl script to do that.

define command {
	command_name    check_j4p_cmd
	command_line    /usr/local/bin/check_jmx4perl --unknown-is-critical --config /etc/jmx4perl/jmx4perl.cfg --server $ARG1$ --check $ARG2$ $ARG3$
}

Then we need to define a service to monitor in /etc/nagios3/conf.d/host-MY_TOMCAT_HOSTNAME.cfg

define service {
	use generic-service
	host_name MY_TOMCAT_HOSTNAME
	service_description Tomcat Heap Memory
	check_command check_j4p_cmd!tomcat!j4p_memory_heap!x
}

define service {
	use generic-service
	host_name MY_TOMCAT_HOSTNAME
	service_description Tomcat AJP Threads
	check_command check_j4p_cmd!tomcat!tc7_connector_threads!ajp-bio-8009
}

The check above is for your tomcat heap and the other one is for Tomcat 7 AJP threads.

Now you should all the pieces to implement your own monitoring using Nagios, Jolokia and JMX4Perl. You should also remember that you can apply this to any JEE application not just Liferay.