Help Docs ~ Server Monititor

Custom Plugins

Put any numeric data you want into Pingdom server monitor with custom plugins.

Suggest Edits

Custom plugins quickstart

A Pingdom server monitor plugin is a Ruby class that runs within Pingdom server monitor to report numerical metrics of your choosing. Your custom plugins get the same first-class treatment as the 80+ plugins in our directory.

This article will guide you through creating a simple Pingdom server monitor plugin.

To get started, ensure the Pingdom server monitor RubyGem is installed on your local development machine: gem install scout

Create the plugin file

  • Create aclass_name.rb file to hold your plugin
  • Build a standard Ruby class that inherits from Scout::Plugin
  • Add a build_report() method that builds up the numerical data you wish to collect

Here's the simplest possible example. Save this as starter_plugin.rb:

class StarterPlugin < Scout::Plugin
  def build_report
    time = Time.new
    report(:hour => time.hour, :minute => time.min)
  end
end

Next, you will run your plug in "test mode" on the command line.

Run your plugin in test mode

From the console on your local development machine:

scout test starter_plugin.rb

You'll see output like this:

== This plugin doesn't have option metadata.
  == You haven't provided any options for running this plugin.
  == Output:
  {:summaries=>[],
   :reports=>
    [{:plugin_id=>nil,
      :created_at=>"2010-02-17 20:21:14",
      :fields=>{:minute=>21, :hour=>12}}],
   :alerts=>[],
   :errors=>[]}

The important bit is the :fields=>{:minute=>21, :hour=>12} -- this corresponds to the hash you provided in report(:hour => time.hour, :minute => time.min).

Run your plugin in Scout

Now that you have the plugin running in "test mode" through the command line, the next step is to place the file on the server your wish to monitor.

  • Copy the plugin to your server via SSH/SCP:
scp starter_plugin.rb my.server.com:~/
  • SSH into your server
  • Ensure the scoutd agent is installed on your server.
  • Place the plugin file in Scout's configuration directory. The default config directory is:
    /var/lib/scoutd.
sudo mv starter_plugin.rb /var/lib/scoutd/
  • Set the scoutd user as the owner/group of the file: 
sudo chown scoutd:scoutd /var/lib/scoutd/starter_plugin.rb

Your new plugin will report the next time the Pingdom server monitor agent runs. You can edit your plugin code as needed. The agent will run the latest version.

You should see a new plugin appear on your server:

Run your plugin in test mode on your server

Sometimes a plugin requires a piece of code, access to a firewalled service, or other functionality that is only available on your monitored server and testing is not possible on your local development machine. In this case, you will need to run the plugin in test mode on the server:

  • Copy the plugin to your server via SSH/SCP:
scp starter_plugin.rb my.server.com:~/
  • Ensure the scoutd agent is installed on your server. Versions 0.5.11 and greater support testing plugins.
  • Place the plugin file in a directory where the scoutd user can access it. If the plugin code contains sensitive information, create a directory readable by only the scoutd user:
sudo mkdir /tmp/scout_test
sudo chown scoutd:scoutd /tmp/scout_test
sudo chmod 700 /tmp/scout_test
sudo mv starter_plugin.rb /tmp/scout_test/
  • Start a bash shell as the scoutd user and run the plugin in test mode:
sudo su - scoutd -s /bin/bash
scoutd test starter_plugin.rb my_plugin_option="somevalue"
  • When you are ready to have the plugin start reporting automatically, move the plugin file to Scout's configuration directory. The default config directory is: /var/lib/scoutd. 
sudo mv starter_plugin.rb /var/lib/scoutd/
  • Make sure to set the scoutd user as the owner/group of the file: 
sudo chown scoutd:scoutd /var/lib/scoutd/starter_plugin.rb

Method reference

Reporting

report(:a => value, :b => value) # report using key-value pairs

Counters

counter(name, value, :per => :second) # reports the rate of change per-second
counter(name, value, :per => :minute) # reports the rate of change per-minute

Memory

remember(:name => value) # remember a value
memory(:key) # retrieve a value from memory

Options

option(:option_name) # access an option

Alerts and errors

alert('subject') # an alert with just a subject
alert('subject','body') # an alert with a subject and body
error('your error text') # an error
error('error subject', 'your error text') # an error with a subject and body

Requiring libraries

needs 'library_name' # load a Ruby library

Running system commands

Many Pingdom server monitor plugins work by running a system command, parsing the output, and reporting its data. Here is a very simple example counting the number of files in your home directory:

class NumFiles < Scout::Plugin
  def build_report
    ls_output = `ls -1 ~`
    num = ls_output.split("\n").size
    report(:num_files=>num)
  end
end

Try creating this file, running it (scout test num_files.rb), and seeing the output.

Plugin options

To make the last example a bit more useful, provide an option for the directory to examine. You specify options with a bit of in-line YAML, like so:

class NumFiles < Scout::Plugin
  # An embedded YAML doc describing the options this plugin takes
  OPTIONS=<<-EOS
    directory:
      default: ~
  EOS

  def build_report
    ls_output = `ls -1 #{option(:directory)}`
    num = ls_output.split("\n").size
    report(:num_files=>num)
  end
end

Note also the option(:directory) in the ls call. This is how you access options provided by the user.

To provide a value for the option at in test mode on your local machine, just do:

scout test num_files.rb directory=/var/log

The existence of the OPTIONS embedded YAML will also generate a corresponding field in the Pingdom server monitor web interface. You can specify any number of options, each containing an option name, a display name, some notes, and a default value.

An example:

OPTIONS=<<-EOS
  directory:
    default: ~
    name: Directory
    notes: The directory in which to count files
EOS

By giving each option a name, and specifying a display name, notes, and a default value, when users add this plugin to their server, they will see the following:

Using Counters

Many system commands return the total count of a metric, but it's often more useful to convert that into a rate. The counter method is designed for this.

Let's say you're retrieving the total number of widgets created in your web application:

total = Widget.count

To report the rate that projects are being created:

counter(:widgets,total, :per => :minute)

The example above will report the rate of widgets created per-minute. You could also report the rate per-second:

counter(:widgets,total, :per => :second)

No data is reported the first time the plugin runs (the initial count needs to be stored in memory).

See the Redis Info Plugin for real-world usage of counters.

Memory

If you need to remember a value between plugin runs, you can remember a value with:

remember :key => value

And retrieve it with:

memory(:key)

See the load averages plugin for an example. This plugin needs to know how many processors your server has in order to properly calculate load averages. Rather than read it from /proc/cpuinfo each time, it uses plugin memory to persist the value.

Note the values stored in plugin memory only last until the next run. If you need to retain the value, you must re-remember it each run. You can persist a value indefinitely like this:

remember(:processors, memory(:processors))

Deploying to the plugin directory

Plugins listed in either the public directory or your private account directory can be installed without placing scripts on your server.

Private Plugins

Installing and maintaining plugin scripts across servers can quickly become a pain. With private plugins, you can simplify this process: publish plugins to your private directory and place a public key on your servers. You'll then be able to install and update plugins without having to login each server.

To get started, create a private-public key pair for your account. You'll sign the plugin code with the private key and place the corresponding public key on your servers. Your servers will then be able to run your signed plugins.

Generate the private-public keypair

On your local development computer, access the Pingdom server monitor config directory (creating it if needed):

mkdir ~/.scout
cd ~/.scout

Inside the .scout directory, generate the private and public keys:

openssl genrsa > scout_rsa
openssl rsa -in scout_rsa -pubout > scout_rsa.pub

The private key ~./scout/scout_rsa is used to sign your custom PSM plugins. Ensure that everyone who creates custom plugins in your organization uses this same private key, and that private key is not available to anyone outside your organization. Keep it safe.

Deploy the public key

Put the public key on every server that uses Pingdom server monitor. The key goes in the Pingdom server monitor config directory: /var/lib/scoutd/scout_rsa.pub and should be owned by the scoutd user. E.g. after uploading the key via SSH:

sudo mv scout_rsa.pub /var/lib/scoutd/scout_rsa.pub
sudo chown scoutd:scoutd /var/lib/scoutd/scout_rsa.pub
sudo scoutctl restart

Placement of the public key on new servers should become part of your provisioning process.

Add the Plugin and sign the code

After clicking the "Add Plugin" button you'll see the plugin directory. In the right column under the "Private Plugins" header, click the "Add" button. Fill in the details for your plugin.

After creating the plugin, you'll be prompted to sign the plugin code. You can run the command from any computer with the Pingdom server monitor private key installed (~/.scout/scout_rsa). The command looks like this:

scout sign http://server.pingdom.com/UNIQUE_PRIVATE_PLUGIN_ID

You're now ready to install the plugin!

Updating Private Plugins

When you change the plugin code, you'll be prompted to run the scout sign command again to update the signature.

Public Plugins

If you've created a plugin that's useful to others, we'd love to add it to our plugin directory. Plugins listed in our public directory are available to everyone and can be installed without placing a file on the server.

To submit your plugin to the public directory, email the url of the private plugin you’d like to share to support.server@pingdom.com. We'll review it for inclusion in the directory.

When to use alerts vs triggers

Alerts are generated by the plugin code itself. Triggers are configured on server.Pingdom.com, and detect changes and thresholds in numerical data reported by plugins. So if you want to generate an alert on a numerical threshold or trend, use Triggers instead of alerts.

Triggers are specified through the web interface at server.pingdom.com and they have a lot of flexibility.

When a trigger fires, an alert is generated for the plugin. There are three kinds of triggers:

  • Peak
  • Plateau
  • Trend

You can configure triggers by clicking on the 'Triggers' menu option when editing a plugin. Note that data must be reported before configuring triggers.

Using ruby libraries and gems

Don't use require or load directly. Instead, use needs, as in:

class MyPlugin < Scout::Plugin
  needs 'mysql2'
  ...

The Scout::Plugin base class defines needs so missing libraries don't break the checkin process.

Note that Rubygems will be loaded if needed to find the libraries you need -- you don't have to explicitly specify Rubygems.

Exceptions

PSM will catch runtime exceptions thrown inside your plugin, and create an Error informing the user of the issue. A rule of thumb: don't catch any exceptions yourself unless it's something the end user can address, i.e., by changing option values. When that's the case, catch the exception and call error("your informative error message") to guide the user.

20 metric limit

Each plugin can send a maximum of 20 unique metrics to Pingdom server monitor. If more than 20 metrics are used, only the first 20 metrics will be tracked. A new metric can't be substituted for one that is no longer used.

Unit tests

What pragmatic plugin developer wouldn't want unit tests? It's easy to build tests for your Pingdom server monitor plugin.

The Test file

Create a file named test.rb in the same directory as the Scout plugin file.

Here's an example test file used to test PSM's HAProxy Plugin:

require File.dirname(__FILE__)+"/../test_helper"
require File.dirname(__FILE__)+"/haproxy_stats"

class HaProxyTestTest < Test::Unit::TestCase
  def test_truth
    assert true # replace me
  end
end

Running a plugin in unit tests

To initialize a plugin: 

plugin=HaproxyStats.new(last_run,memory = {},options = {})

last_run is a Time object that indicates the last time the plugin ran. Set this to nil to simulate the first run of the plugin. This time is accessible in the plugin as @last_run.

memory is a Hash that contains any memory settings for the plugin. For example, you might store the value of a metric in a previous run of a plugin and calculate how the metric has changed in the next run. Learn how to access memory in a plugin.

options is a Hash that contains any options for the plugin. For example, you might need an option to specify the port used to access an Apache status page. Learn how to access options in a plugin.

To run a plugin, call the Plugin#run method:

# initialize
plugin=HaproxyStats.new(nil) # no last run, no memory, and no options
# run!
result = plugin.run
Plugin#run returns a Hash. Here's a sample:

{# when the plugin ran, it stored this in memory for access in the next run.
 :memory=>{:stored_disk_usage => 30}, 
 # An Array of report Hashes
 :reports=>[{:disk_size=>38.0, :memory_usage=>'691 KB'}], 
 # An Array of Alerts, each is a Hash
 :alerts=>[{:subject=>"Memory Usage Alert", 
            :body=>"Memory usage has exceeded 95%. Memory: 691KB."}],
 # Similar to Alerts
 :errors=>[]
}

Example

plugin = LoadAverages.new(Time.now-5*60,{:previous_load => 1.0},
                          {:num_processors => 1})
result = plugin.run
assert_equal 2.0 result[:reports].first[:cpu_last_minute]
assert_equal 2.0 result[:memory][:previous_load]
assert_equal 'Load Increased', result[:alerts].first[:subject]

Running tests

$ ruby test.rb 
Loaded suite test
Started
.....
Finished in 0.492469 seconds.

5 tests, 14 assertions, 0 failures, 0 errors

Advanced examples

Take a look at the following tests for Scout Plugins available on Github:

Alerts and errors

Alerts

If you'd like to generate an alert from within your plugin, just call:

alert('subject')

Or if you've got more to say, include an alert body:

alert('subject','body')

Know when to use an alert or trigger: see Alerts vs triggers above.

Errors

You can also create error messages. You should use errors when something has prevented your plugin from operating properly and the user can do something about it. A good example of an error is a missing gem dependency.

Generate an error with:

error('your error text')

Errors can also have a subject and body:

error('error subject','your error text')

Know when to use an error or exception: Exceptions

Custom Plugins

Put any numeric data you want into Pingdom server monitor with custom plugins.