Now you might wonder why would you need MRTG if you already have Nagios. Nagios operates on the present value although there’s a add-on nagios grapher that can create graphs like MRTG does but I like MRTG more because you can see all the graphs on one page. Being able to see a full overview of the system is very important when trying to identify performance problems. This is also why you want to get more information out of the application, Liferay in this case. My script will help in reading connection pool and thread pool utilization as well as heap usage. Those are essential when doing troubleshooting.

First you’ll need to install and setup MRTG. I’m not going to go into details on that because it dependes on your system and the internet is full of guides to do it. Once you have it done you’ll need to download my mrtg-jmx4perl.pl script which is available in my github repository. For rest of this post I’m going to assume it’s located in /usr/local/bin/mrtg-jmx4perl.pl but it’s up to you where you put it. Just adjust the script path accordingly.

Monitoring c3p0 connection pool

Getting the values for c3p0 is a little bit tricky because it will generate a unique mbean name based on the identity token it generates for the connection pool every time the server is started. Because of this my script assumes you only have one c3p0 connection pool if you have multiple pool you’ll need to add additional logic in to the mrtg-jmx4perl to find the correct mbean. Notice that this is the case if you configure Liferay to use connection pool from portal-ext.properties instead of using a JNDI resource. We can read the mbean for c3p0 by using mbean name as “com.mchange.v2.c3p0:type=*,*” and the attributes we are most interested are numConnectionsAllUsers and numBusyConnectionsAllUsers. Below is a sample mrtg configuration snippet.

Target[dbpool]: `/usr/local/bin/mrtg-jmx4perl.pl --server=servername --mbean="com.mchange.v2.c3p0:type=*,*" --attribute="numConnectionsAllUsers numBusyConnectionsAllUsers"`
MaxBytes[dbpool]: 20
Title[dbpool]: DB Pool
PageTop[dbpool]: <h1>DB Pool</h1>
WithPeak[dbpool]: dwmy
Unscaled[dbpool]: dwmy
Options[dbpool]: growright,unknaszero,nopercent,gauge
YLegend[dbpool]: Connections
ShortLegend[dbpool]: 
LegendI[dbpool]: Connections
LegendO[dbpool]: Busy Connections
Legend1[dbpool]: Connections
Legend2[dbpool]: Busy Connections
Legend3[dbpool]: Peak Connections
Legend4[dbpool]: Peak Busy Connections

Here’s a daily graph from one of my Liferay portal servers.

Monitoring Tomcat AJP Thread Pool

This one is pretty easy because the the mbean name is static but it does vary depending on do tomcat version and connector you are using. In Tomcat 7 with native library the name for ajp thread pool is Catalina:type=ThreadPool,name=”ajp-apr-8009″. Without native library it would be ajp-bio-8009. In tomcat 6 my ajp pool mbean name is Catalina:type=ThreadPool,name=jk-8009. Notice the lack of double quotes in the name. You can easily check the name using jconsole. So for this one the config looks like:

Target[ajp-threadpool]: `/usr/local/bin/mrtg-jmx4perl.pl --server=servername --mbean="Catalina:type=ThreadPool,name=\"ajp-apr-8009\"" --attribute="currentThreadCount currentThreadsBusy"`
MaxBytes[ajp-threadpool]: 50
Title[ajp-threadpool]: AJP Thread Pool
PageTop[ajp-threadpool]: <h1>AJP Thread Pool</h1>
WithPeak[ajp-threadpool]: dwmy
#Unscaled[ajp-threadpool]: dwmy
Options[ajp-threadpool]:  growright,unknaszero,nopercent,gauge
YLegend[ajp-threadpool]: Threads
ShortLegend[ajp-threadpool]: 
LegendI[ajp-threadpool]: Threads
LegendO[ajp-threadpool]: Busy Threads
Legend1[ajp-threadpool]: Threads
Legend2[ajp-threadpool]: Busy Threads
Legend3[ajp-threadpool]: Peak Threads
Legend4[ajp-threadpool]: Peak Busy Threads

Here’s a daily graph of a thread pool.

Monitoring Heap Usage

The last one we are going to monitor is Java Heap usage. It can be read from java.lang:type=Memory using attribute HeapMemoryUsage and path used. Now we are reading only one value.

Target[heap]: `/usr/local/bin/mrtg-jmx4perl.pl --server=servername --mbean="java.lang:type=Memory" --attribute="HeapMemoryUsage" --path="used"`
MaxBytes[heap]: 1296302080
Title[heap]: Heap
PageTop[heap]: <h1>Heap</h1>
WithPeak[heap]: dwmy
Unscaled[heap]: dwmy
Options[heap]:  growright,unknaszero,nopercent,gauge,noo
YLegend[heap]: bytes
ShortLegend[heap]: 
kilo[heap]: 1024
LegendI[heap]: Used
Legend1[heap]: Used
Legend3[heap]: Peak Used

Here’s a daily graph of heap memory usage.

You can download the full sample-mrtg.cfg from github.

That’s how easy it is to monitor and graph Liferay or pretty much any Java webapp using MRTG. You could easily use this to monitor ehcache utilization or anything else that’s accessible via JMX.

There’s a nice agent called Jolokia that can provide a http bridge to JMX. You can install it as java agent in pretty much any java app or deploy it as a webapp. With Jolokia installed you can query any MBeans for their values using a simple http GET and get the data as JSON objects. JMX4Perl is a perl module and scripts that provide a easy way to run those queries through Jolokia. One of those scripts is check_jmx4perl which can be used in nagios service checks.

Okay so now we know that we are going to need Nagios, Jolokia and JMX4Perl to monitor the Liferay JVM but what should we monitor? Well that depends on what information you are interested in but at minimum I would monitor ajp or http thread usage as well as heap utilization. Just by monitoring those values you’ll know when your JVM is becomes unresponsive and can also get some early warning that there’s issues for example heap usage goes over warning threashold and never returns to normal or keeps constantly going over the threshold which could indicate they you don’t have enough heap allocated.

I’m going to assume that you have  nagios installed and configured and I will only go through how to install Jolokia and configure some checks for threads and heap. So let’s start by installing JMX4Perl.

Installing JMX4Perl is pretty simple with cpan. You just launch cpan command line client and install it like this:

cpan> install JMX::Jmx4Perl

Next you’ll need to download Jolokia and deploy the jolokia.war to your app server. For this example I’m going to assume that you are using Tomcat 7. Once you’ve deployed Jolokia it’s usually good idea to restrict who can query it. For this example we are just going to restrict it to a certain IP address (the Nagios server) and limit it to read operations only. Since I don’t like modifying the war we are going to tell Jolokia where to find the policy file through a context parameter. Create a jolokia.xml in tomcat/conf/Catalina/localhost with following content:

<Context path="/jolokia">
        <Parameter name="policyLocation" value="file:///etc/jolokia/jolokia-access.xml" />
</Context>

That tells Jolokia to look for the policy file jolokia-access.xml from /etc/jolokia/jolokia-access.xml. This is great when you are running multiple tomcats in the same server and want them to share the jolokia policy file.

Now go ahead and create the jolokia-access.xml in /etc/jolokia

<?xml version="1.0" encoding="utf-8"?>
<restrict>
        <remote>
                <host>[YOUR NAGIOS SERVER IP]</host>
        </remote>
        <http>
                <method>get</method>
                <method>post</method>
        </http>
        <commands>
                <command>read</command>
        </commands>
</restrict>

Next we need to create configuration for jmx4perl. In /etc/jmx4perl/jmx4perl.cfg we are going to include some preconfigured checks extend them. Tomcat 7 you need to add quotes around the thread pool name. We also need to set warning and critical levels for alerts. You’ll also need to add a Server for each tomcat you want to monitor.

# Default definitions
include default/memory.cfg
include default/tomcat.cfg

# ==========================
# Check definitions

<Check tc7_connector_threads>
	Use = relative_base($1,$2)
	Label = Connector $0 : $BASE
	Value = Catalina:name="$0",type=ThreadPool/currentThreadCount
	Base = Catalina:name="$0",type=ThreadPool/maxThreads
	Critical 95
	Warning 90
</Check>

<Check j4p_memory_heap>
	Use memory_heap
	Critical 95
	Warning 90
</Check>

<Server tomcat>
	Url http://MY_TOMCAT_HOSTNAME:8080/jolokia
</Server>

Then in /etc/nagios3/commands.cfg we’ll need to add a check command for jmx4perl and we’ll use the check_jmx4perl script to do that.

define command {
	command_name    check_j4p_cmd
	command_line    /usr/local/bin/check_jmx4perl --unknown-is-critical --config /etc/jmx4perl/jmx4perl.cfg --server $ARG1$ --check $ARG2$ $ARG3$
}

Then we need to define a service to monitor in /etc/nagios3/conf.d/host-MY_TOMCAT_HOSTNAME.cfg

define service {
	use generic-service
	host_name MY_TOMCAT_HOSTNAME
	service_description Tomcat Heap Memory
	check_command check_j4p_cmd!tomcat!j4p_memory_heap!x
}

define service {
	use generic-service
	host_name MY_TOMCAT_HOSTNAME
	service_description Tomcat AJP Threads
	check_command check_j4p_cmd!tomcat!tc7_connector_threads!ajp-bio-8009
}

The check above is for your tomcat heap and the other one is for Tomcat 7 AJP threads.

Now you should all the pieces to implement your own monitoring using Nagios, Jolokia and JMX4Perl. You should also remember that you can apply this to any JEE application not just Liferay.