Drilling down to a single server overview shows request velocity, cpu load, busy/idle workers and even bytes sent over time.

Going to throughtput shows throughtput details over time.

Workers section shows you what is happening with the workers. If you have a lot of busy workers you can see in what state their are and that might provide some insight to what is going on.

Installing Apache HTTPd agent for New Relic

1) I’m using the MeetMe New Relic Agent to monitor Apache HTTPd. It’s written in python and we’ll need to install pip. The following is using the Ubuntu python-pip package. You can find alternative install methods from pip docs.

apt-get install python-pip libyaml-dev python-dev

2) Next use pip to install newrelic-plugin-agent. When I ran it I got some errors but it still worked.

pip install newrelic-plugin-agent

3) Next we’ll need to create the configuration file for the agent. You can start by copying /opt/newrelic-plugin-agent/newrelic-plugin-agent.cfg or just use what I have posted below. The first thing you need is to set license_key. You can find your license key from your account settings page on rpm.newrelic.com. The second one is to add apache_httpd configuration. You can add multiple httpds to monitor.

cat - <<EOF>> /etc/newrelic/newrelic-plugin-agent.cfg
%YAML 1.2
---
Application:
 license_key: YOUR_LICENSE_KEY
 wake_interval: 60
 #newrelic_api_timeout: 10
 #proxy: http://localhost:8080

 apache_httpd:
  - name: localhost
    scheme: http
    host: localhost
    verify_ssl_cert: true
    port: 80
    path: /server-status

Daemon:
 user: newrelic
 pidfile: /var/run/newrelic/newrelic-plugin-agent.pid

Logging:
 formatters:
   verbose:
     format: '%(levelname) -10s %(asctime)s %(process)-6d %(processName) -15s %(threadName)-10s %(name) -45s %(funcName) -25s L%(lineno)-6d: %(message)s'
 handlers:
   file:
     class : logging.handlers.RotatingFileHandler
     formatter: verbose
     filename: /var/log/newrelic/newrelic-plugin-agent.log
     maxBytes: 10485760
     backupCount: 3
 loggers:
   newrelic_plugin_agent:
     level: INFO
     propagate: True
     handlers: [console, file]
   requests:
     level: ERROR
     propagate: True
     handlers: [console, file]
EOF

4) Make sure you have enabled mod_status in your apache and you’ve allowed access from the host your agent is running if it’s not running in the same host as your apache.

5) Then we need to add a init script for the newrelic-plugin-agent. There’s one under /opt/newrelic-plugin-agent but for me it was incomplete file so I just downloaded the one from github.

wget https://raw.githubusercontent.com/MeetMe/newrelic-plugin-agent/master/etc/init.d/newrelic-plugin-agent.deb
mv newrelic-plugin-agent.deb /etc/init.d/newrelic-plugin-agent
chmod 755 /etc/init.d/newrelic-plugin-agent
update-rc.d newrelic-plugin-agent defaults

Now you can start the newrelic-plugin-agent with

service newrelic-plugin-agent start

Now in few minutes you should see your Apache HTTPd server(s) listed under Plugins HTTPd on rpm.newrelic.com.

From the overview you can see the number of active and idle connections as well as the request rate.

From connections you’ll even more connection details. With very little connections and requests my graphs are currently slightly boring. In addition to connection details you can find more details about requests, upstreams, servers and cache.

Installing New Relic Monitoring Agent for Nginx

1) First you need to add the Ubuntu package repository for Nginx. If you’ve done this already when you installed Nginx you can skip to next step. If you are not using Ubuntu 14.04 like I am you can find the other Linux packages from Nginx website.

wget http://nginx.org/keys/nginx_signing.key
apt-key add nginx_signing.key

cat - <<-EOF >> /etc/apt/sources.list.d/nginx.list
deb http://nginx.org/packages/ubuntu/ trusty nginx
deb-src http://nginx.org/packages/ubuntu/ trusty nginx
EOF

apt-get update

2) Next you need to install the Nginx New Relic Agent

apt-get install nginx-nr-agent

3) Next you’ll need to edit the agent configuration file in /etc/nginx-nr-agent/nginx-nr-agent.ini. You need to add your license key which you can find from your account settings page on rpm.newrelic.com.

newrelic_license_key=YOUR_LICENSE_KEY

Additionally you need to add a new source which points to your Nginx status url.

[source1]
name=localhost
url=http://localhost/nginx_stub_status

4) You’ll need to add a server block to Nginx for the status. Since I had very simple configuration in my Nginx I just added the following to /etc/nginx/sites-enabled/default

server {
   listen 127.0.0.1:80;
   server_name localhost;

   location = /nginx_stub_status {
     stub_status on;
     allow 127.0.0.1;
     deny all;
   }
}

5) Last thing you need to do is reload Nginx and start the Nginx New Relic Agent.

service nginx reload
service nginx-nr-agent start

Now in few minutes you should start seeing your Nginx server listed under Plugins Nginx on rpm.newrelic.com.

New Relic has a wonderful plugin framework and there’s a ton of ready made plugins and also SDK and API for things it doesn’t already support. MySQL plugin is one of those ready made plugins and it provides all the key information you’ll need. The MySQL plugin page quickly shows what’s going on all monitored databases.

When you drill down to a individual database server the overview shows the SQL volume and how it’s split between reads and writes. More key metrics are displayed under Key Utilizations. You’ll also find database connections and network traffic on this page.

Going further down to Query analysis you’ll see in more details about the queries.

If you are using InnoDB there’s a separate page to show key metrics from InnoDB.

Installing MySQL / MariaDB Monitoring

1) MySQL plugin can easily be installed with New Relic platform installer. So the first thing you need to do is install the platform installer. You’ll need your New Relic license key which you can find from account settings on rpm.newrelic.com. Once you have that you can install it with following one liner which is for 64bit Debian and Ubuntu.

LICENSE_KEY=YOUR_LICENSE_KEY bash -c "$(curl -sSL https://download.newrelic.com/npi/release/install-npi-linux-debian-x64.sh)"

2) Next go to the newly created newrelic-npi directory and run install. You’ll want to answer yes to all the questions and when prompted to configure the plugin grab the configuration from the next step.

./npi install nrmysql

3) If you skipped configuration you can configure the plugin afterward too. You can find the configuration file under newrelic-npi from plugins/com.newrelic.plugins.mysql.instance/newrelic_mysql_plugin-2.0.0/config/plugin.json. Below is a sample configuration for MariaDB (works for MySQL) running on localhost and we’ll be creating a separate user newrelic with password somepassword which the plugin will use to gather data. You can connect to multiple databases with the same agent. I usually install this agent on the same server my nagios is running on.

{
 "agents": [
   {
     "name" : "MariaDB on localhost",
     "host" : "localhost",
     "metrics" : "status,newrelic,buffer_pool_stats,innodb_status,innodb_mutex",
     "user" : "newrelic",
     "passwd" : "somepassword"
   }
 ]
}

4) Now we need to create a user in the database and grant some rights to it.

cat - <<EOF | mysql -u root -p
CREATE USER newrelic@'%' IDENTIFIED BY 'somepassword';
GRANT PROCESS,REPLICATION CLIENT ON *.* TO newrelic@'%';
EOF

5) Last thing is to start the service but before we do make sure you have Java installed as this agent is written in Java. If you don’t have Java installed check my unattended Java install script. Otherwise you can proceed to start the service that should have been created during npi install if you answered all the questions correctly.

service newrelic_plugin_com.newrelic.plugins.mysql.instance start

Now it may take few minutes before you see your server under Plugins MySQL in rpm.newrelic.com. If it doesn’t check the log under plugins/com.newrelic.plugins.mysql.instance/newrelic_mysql_plugin-2.0.0/logs/ for hints and make sure the agent actually started.

The servers listing gives a nice overview of all servers and you an easily see if there’s any issues.

When looking at a specific server you’ll see a history of it’s CPU and memory usage as well as load average and network I/O. If you have any APM enabled applications installed you’ll see a overview of their response times, throughput and error rate. You’ll also see some of the top processes running on the server.

When you drill down to processes listing you’ll quickly see the top memory and cpu consumers. You can also look at the history of individual processes.

Installing New Relic Server Monitoring on a Ubuntu / Debian Server

1) Add an apt source for New Relic.

cat - <<-EOF >> /etc/apt/sources.list.d/newrelic.list 
# newrelic repository list 
deb http://apt.newrelic.com/debian/ newrelic non-free
EOF

2) You’ll need to get the key for New Relic repository and then update apt sources. After that you can install newrelic-sysmond.

apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 0xB31B29E5548C16BF
apt-get update
apt-get install newrelic-sysmond

3) Next you’ll need to tell it your license key so that it reports the data to your account. You can find your license key from your account settings page on rpm.newrelic.com. You can either edit the configuration file or you can set the license like shown below:

/usr/sbin/nrsysmond-config --set license_key=YOUR_LICENSE_KEY

4) Finally once everything is configured you can start the system monitor daemon.

service newrelic-sysmond start

Now in few minutes you should start seeing your server listed under Servers on rpm.newrelic.com

New Relic has seven parts or products as they call them. Those are APM, Insights, Mobile, Browser, Synthetics, Servers and Plugins. I have myself used only APM, Browser, Server and Plugins which are included in the free Lite edition.

APM

APM is the application monitoring part. It focuses on providing information about the application itself. The Lite edition shows you response times, throughput, web transaction information. It’s basically a low impact profiler. With the Pro subscription you get much deeper analysis of time spent on executing SQL, JVM statistics etc.

Browser

The browser provides insights on client side performance. Even though your application might respond quickly the users perceived performance could be poor because of network performance or even how the page is rendering on the browser.

Servers

Servers as the name suggests provides performance information about the actual server your applications are running on.

Plugins

There’s a ton of plugins to provide monitoring capabilities to systems not otherwise supported by New Relic and with it’s SDKs and API you can build your own plugins. Some of the plugins I have used are for MySQL/MariaDB, Nginx and Apache.

Mobile

Mobile is APM for mobile applications.

Synthetics

Synthetics allows you to test your application from around the world. It can check business critical user flows and interaction to make sure your site available and functioning from around the world.

Insights

Insights is a paid feature that combines business metrics with performance data. It can combine data from APM, Browser, Mobile and Synthetics for deeper analysis and segmentation and filtering.

Since I recently installed bunch of new servers and I had to refresh my memory on how I installed and configured each of the agents so I decided to write a series of articles on each of them. Here’s a list of topics I’m going to publish and as I publish them I’ll link the topic to the article. These topics will cover APM with Java and PHP, Servers and Plugins for MySQL, Nginx and Apache

• Monitoring Ubuntu / Debian Servers
• Monitoring MariaDB / MySQL
• Monitoring Nginx
• Monitoring Apache
• Monitoring Tomcat
• Monitoring PHP Applications