It’s very common to terminate HTTPS (TLS) at higher up in your server stack but you still need your webapp running in tomcat to generate the urls using https even though tomcat is called with http internally in your network. This seems to be a very common problem that I keep seeing year after year so this short article will show you how to accomplish that and how to test it’s working.

In this diagram https is terminated at firewall but that could as well be loadbalancer or even http server like Nginx or Apache. For the test setup I’m actually using Nginx and for instructions on how to setup HTTPS with Nginx check out my post on setting up Nginx with Let’s Encrypt. Once you’ve setup https with Nginx add following location block to the server block with HTTPS. This will proxy all requests to tomcat http port 8080.

location ~ / {
      proxy_set_header   Host             $host;
      proxy_set_header   X-Real-IP        $remote_addr;
      proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
      proxy_set_header   REMOTE_ADDR      $remote_addr;

      proxy_pass         http://localhost:8080;
}

Tomcat is actually really easy to configure so that it is able to generate URLs with https when the https is terminated somewhere higher in the stack. All you need is to add proxyPort, scheme, and secure to the connector in server.xml. Below is an example. If you are using both http and https then just create a new connector for https traffic that is running in different port and proxy only the https traffic to that port.

<Connector port="8080" protocol="HTTP/1.1"
      connectionTimeout="20000"
      redirectPort="8443" 
      proxyPort="443"
      scheme="https"
      secure="true" />

Now to check that it is actually working correctly you need to check your servlet container is seeing those values correctly. For that purpose I’ve created a simple webapp which you can deploy and call through your stack. It shows if each of the three checks pass and additionally shows the request URL and server name so you can also check that any virtual hosts you use are also passed correctly to the servlet container.

Download HTTPS Checker webapp. See the source in Github.

There’s a nice agent called Jolokia that can provide a http bridge to JMX. You can install it as java agent in pretty much any java app or deploy it as a webapp. With Jolokia installed you can query any MBeans for their values using a simple http GET and get the data as JSON objects. JMX4Perl is a perl module and scripts that provide a easy way to run those queries through Jolokia. One of those scripts is check_jmx4perl which can be used in nagios service checks.

Okay so now we know that we are going to need Nagios, Jolokia and JMX4Perl to monitor the Liferay JVM but what should we monitor? Well that depends on what information you are interested in but at minimum I would monitor ajp or http thread usage as well as heap utilization. Just by monitoring those values you’ll know when your JVM is becomes unresponsive and can also get some early warning that there’s issues for example heap usage goes over warning threashold and never returns to normal or keeps constantly going over the threshold which could indicate they you don’t have enough heap allocated.

I’m going to assume that you have  nagios installed and configured and I will only go through how to install Jolokia and configure some checks for threads and heap. So let’s start by installing JMX4Perl.

Installing JMX4Perl is pretty simple with cpan. You just launch cpan command line client and install it like this:

cpan> install JMX::Jmx4Perl

Next you’ll need to download Jolokia and deploy the jolokia.war to your app server. For this example I’m going to assume that you are using Tomcat 7. Once you’ve deployed Jolokia it’s usually good idea to restrict who can query it. For this example we are just going to restrict it to a certain IP address (the Nagios server) and limit it to read operations only. Since I don’t like modifying the war we are going to tell Jolokia where to find the policy file through a context parameter. Create a jolokia.xml in tomcat/conf/Catalina/localhost with following content:

<Context path="/jolokia">
        <Parameter name="policyLocation" value="file:///etc/jolokia/jolokia-access.xml" />
</Context>

That tells Jolokia to look for the policy file jolokia-access.xml from /etc/jolokia/jolokia-access.xml. This is great when you are running multiple tomcats in the same server and want them to share the jolokia policy file.

Now go ahead and create the jolokia-access.xml in /etc/jolokia

<?xml version="1.0" encoding="utf-8"?>
<restrict>
        <remote>
                <host>[YOUR NAGIOS SERVER IP]</host>
        </remote>
        <http>
                <method>get</method>
                <method>post</method>
        </http>
        <commands>
                <command>read</command>
        </commands>
</restrict>

Next we need to create configuration for jmx4perl. In /etc/jmx4perl/jmx4perl.cfg we are going to include some preconfigured checks extend them. Tomcat 7 you need to add quotes around the thread pool name. We also need to set warning and critical levels for alerts. You’ll also need to add a Server for each tomcat you want to monitor.

# Default definitions
include default/memory.cfg
include default/tomcat.cfg

# ==========================
# Check definitions

<Check tc7_connector_threads>
	Use = relative_base($1,$2)
	Label = Connector $0 : $BASE
	Value = Catalina:name="$0",type=ThreadPool/currentThreadCount
	Base = Catalina:name="$0",type=ThreadPool/maxThreads
	Critical 95
	Warning 90
</Check>

<Check j4p_memory_heap>
	Use memory_heap
	Critical 95
	Warning 90
</Check>

<Server tomcat>
	Url http://MY_TOMCAT_HOSTNAME:8080/jolokia
</Server>

Then in /etc/nagios3/commands.cfg we’ll need to add a check command for jmx4perl and we’ll use the check_jmx4perl script to do that.

define command {
	command_name    check_j4p_cmd
	command_line    /usr/local/bin/check_jmx4perl --unknown-is-critical --config /etc/jmx4perl/jmx4perl.cfg --server $ARG1$ --check $ARG2$ $ARG3$
}

Then we need to define a service to monitor in /etc/nagios3/conf.d/host-MY_TOMCAT_HOSTNAME.cfg

define service {
	use generic-service
	host_name MY_TOMCAT_HOSTNAME
	service_description Tomcat Heap Memory
	check_command check_j4p_cmd!tomcat!j4p_memory_heap!x
}

define service {
	use generic-service
	host_name MY_TOMCAT_HOSTNAME
	service_description Tomcat AJP Threads
	check_command check_j4p_cmd!tomcat!tc7_connector_threads!ajp-bio-8009
}

The check above is for your tomcat heap and the other one is for Tomcat 7 AJP threads.

Now you should all the pieces to implement your own monitoring using Nagios, Jolokia and JMX4Perl. You should also remember that you can apply this to any JEE application not just Liferay.

The first thing is to copy or move the c3p0.jar from webapps/ROOT/WEB-INF/lib/ to lib/. Also make sure you have your dabase driver there. In this example it would be mysql.jar.

Then we need to tell Liferay that you want to use a connection pool from JNDI and this you can do by adding following line to your portal-ext.properties which can be placed in Liferay Home directory (the directory above tomcat).

jdbc.default.jndi.name=jdbc/LiferayPool

Add following snippet to conf/server.xml inside GlobalNamingResources. Adjust the pool size and idle time and connection test period according to your environment. They are particularly important when you have a firewall between your Liferay and database or when the database server drops connections after certain idle period.

<Resource
    name="jdbc/LiferayPool"
    auth="Container"
    type="com.mchange.v2.c3p0.ComboPooledDataSource"
    factory="org.apache.naming.factory.BeanFactory"
    driverClass="com.mysql.jdbc.Driver"
    jdbcUrl="jdbc:mysql://localhost/lportaluseUnicode=true&amp;characterEncoding=UTF-8&amp;useFastDateParsing=false"
    user="lportal"
    password="test"
    minPoolSize="10"
    maxPoolSize="20"
    maxIdleTime="600"
    preferredTestQuery="select 1 from dual"
    idleConnectionTestPeriod="180"
    numHelperThreads="5"
    maxStatementsPerConnection="100"
/>

Now we need to link the jdbc/LiferayPool name defined in portal-ext.properties to the jdbc/LiferayPool defined in server.xml and this definition goes to conf/Catalina/localhost/ROOT.xml

<ResourceLink name="jdbc/LiferayPool" global="jdbc/LiferayPool" type="javax.sql.DataSource"/>

Now we are done and you can start your tomcat with the new connection pool. Note you can follow similar process to configure MailSession from JNDI.

SEVERE: Error listenerStart 
26-May-2012 13:44:27 org.apache.catalina.core.StandardContext startInternal 
SEVERE: Context [] startup failed due to previous errors

That basically means that tomcat failed to start the webapp because there was an error with some listener, quite often Spring context listener. The really annoying part is that it doesn’t actually show you what went wrong. There’s actually pretty simple way to get tomcat to log the actual error. You just need to create a logging.properties in WEB-INF/classes of the failing webapp and add following lines to it:

org.apache.catalina.core.ContainerBase.[Catalina].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].handlers = java.util.logging.ConsoleHandler

Then just reload the webapp to see the error in tomcat console log. I hope this tip saves you a lot of hasle from trying to figure out the root cause of the problem.