A Pingdom server monitor plugin is a Ruby class that runs within Pingdom server monitor to report numerical metrics of your choosing. Your custom plugins get the same first-class treatment as the 80+ plugins in our directory.
This article will guide you through creating a simple Pingdom server monitor plugin.
To get started, ensure the Pingdom server monitor RubyGem is installed on your local development machine: gem install scout
Create the plugin file
- Create a
class_name.rb
file to hold your plugin - Build a standard Ruby class that inherits from
Scout::Plugin
- Add a
build_report()
method that builds up the numerical data you wish to collect
Here's the simplest possible example. Save this as starter_plugin.rb
:
class StarterPlugin < Scout::Plugin
def build_report
time = Time.new
report(:hour => time.hour, :minute => time.min)
end
end
Next, you will run your plug in "test mode" on the command line.
Run your plugin in test mode
From the console on your local development machine:
scout test starter_plugin.rb
You'll see output like this:
== This plugin doesn't have option metadata.
== You haven't provided any options for running this plugin.
== Output:
{:summaries=>[],
:reports=>
[{:plugin_id=>nil,
:created_at=>"2010-02-17 20:21:14",
:fields=>{:minute=>21, :hour=>12}}],
:alerts=>[],
:errors=>[]}
The important bit is the :fields=>{:minute=>21, :hour=>12}
-- this corresponds to the hash you provided in report(:hour => time.hour, :minute => time.min)
.
Run your plugin in Scout
Now that you have the plugin running in "test mode" through the command line, the next step is to place the file on the server your wish to monitor.
- Copy the plugin to your server via SSH/SCP:
scp starter_plugin.rb my.server.com:~/
- SSH into your server
- Ensure the scoutd agent is installed on your server.
- Place the plugin file in Scout's configuration directory. The default config directory is:
/var/lib/scoutd
.
sudo mv starter_plugin.rb /var/lib/scoutd/
- Set the scoutd user as the owner/group of the file:
sudo chown scoutd:scoutd /var/lib/scoutd/starter_plugin.rb
Your new plugin will report the next time the Pingdom server monitor agent runs. You can edit your plugin code as needed. The agent will run the latest version.
You should see a new plugin appear on your server:
Run your plugin in test mode on your server
Sometimes a plugin requires a piece of code, access to a firewalled service, or other functionality that is only available on your monitored server and testing is not possible on your local development machine. In this case, you will need to run the plugin in test mode on the server:
- Copy the plugin to your server via SSH/SCP:
scp starter_plugin.rb my.server.com:~/
- Ensure the scoutd agent is installed on your server. Versions 0.5.11 and greater support testing plugins.
- Place the plugin file in a directory where the
scoutd
user can access it. If the plugin code contains sensitive information, create a directory readable by only thescoutd
user:
sudo mkdir /tmp/scout_test
sudo chown scoutd:scoutd /tmp/scout_test
sudo chmod 700 /tmp/scout_test
sudo mv starter_plugin.rb /tmp/scout_test/
- Start a bash shell as the
scoutd
user and run the plugin in test mode:
sudo su - scoutd -s /bin/bash
scoutd test starter_plugin.rb my_plugin_option="somevalue"
- When you are ready to have the plugin start reporting automatically, move the plugin file to Scout's configuration directory. The default config directory is:
/var/lib/scoutd
.
sudo mv starter_plugin.rb /var/lib/scoutd/
- Make sure to set the scoutd user as the owner/group of the file:
sudo chown scoutd:scoutd /var/lib/scoutd/starter_plugin.rb
report(:a => value, :b => value) # report using key-value pairs
counter(name, value, :per => :second) # reports the rate of change per-second
counter(name, value, :per => :minute) # reports the rate of change per-minute
remember(:name => value) # remember a value
memory(:key) # retrieve a value from memory
option(:option_name) # access an option
alert('subject') # an alert with just a subject
alert('subject','body') # an alert with a subject and body
error('your error text') # an error
error('error subject', 'your error text') # an error with a subject and body
needs 'library_name' # load a Ruby library
Many Pingdom server monitor plugins work by running a system command, parsing the output, and reporting its data. Here is a very simple example counting the number of files in your home directory:
class NumFiles < Scout::Plugin
def build_report
ls_output = `ls -1 ~`
num = ls_output.split("\n").size
report(:num_files=>num)
end
end
Try creating this file, running it (scout test num_files.rb
), and seeing the output.
To make the last example a bit more useful, provide an option for the directory to examine. You specify options with a bit of in-line YAML, like so:
class NumFiles < Scout::Plugin
# An embedded YAML doc describing the options this plugin takes
OPTIONS=<<-EOS
directory:
default: ~
EOS
def build_report
ls_output = `ls -1 #{option(:directory)}`
num = ls_output.split("\n").size
report(:num_files=>num)
end
end
Note also the option(:directory)
in the ls call. This is how you access options provided by the user.
To provide a value for the option at in test mode on your local machine, just do:
scout test num_files.rb directory=/var/log
The existence of the OPTIONS embedded YAML will also generate a corresponding field in the Pingdom server monitor web interface. You can specify any number of options, each containing an option name, a display name, some notes, and a default value.
An example:
OPTIONS=<<-EOS
directory:
default: ~
name: Directory
notes: The directory in which to count files
EOS
By giving each option a name, and specifying a display name, notes, and a default value, when users add this plugin to their server, they will see the following:
Many system commands return the total count of a metric, but it's often more useful to convert that into a rate. The counter
method is designed for this.
Let's say you're retrieving the total number of widgets created in your web application:
total = Widget.count
To report the rate that projects are being created:
counter(:widgets,total, :per => :minute)
The example above will report the rate of widgets created per-minute. You could also report the rate per-second:
counter(:widgets,total, :per => :second)
No data is reported the first time the plugin runs (the initial count needs to be stored in memory).
See the Redis Info Plugin for real-world usage of counters.
If you need to remember a value between plugin runs, you can remember a value with:
remember :key => value
And retrieve it with:
memory(:key)
See the load averages plugin for an example. This plugin needs to know how many processors your server has in order to properly calculate load averages. Rather than read it from /proc/cpuinfo
each time, it uses plugin memory to persist the value.
Note the values stored in plugin memory only last until the next run. If you need to retain the value, you must re-remember it each run. You can persist a value indefinitely like this:
remember(:processors, memory(:processors))
Plugins listed in either the public directory or your private account directory can be installed without placing scripts on your server.
Private Plugins
Installing and maintaining plugin scripts across servers can quickly become a pain. With private plugins, you can simplify this process: publish plugins to your private directory and place a public key on your servers. You'll then be able to install and update plugins without having to login each server.
To get started, create a private-public key pair for your account. You'll sign the plugin code with the private key and place the corresponding public key on your servers. Your servers will then be able to run your signed plugins.
Generate the private-public keypair
On your local development computer, access the Pingdom server monitor config directory (creating it if needed):
mkdir ~/.scout
cd ~/.scout
Inside the .scout
directory, generate the private and public keys:
openssl genrsa > scout_rsa
openssl rsa -in scout_rsa -pubout > scout_rsa.pub
The private key ~./scout/scout_rsa
is used to sign your custom PSM plugins. Ensure that everyone who creates custom plugins in your organization uses this same private key, and that private key is not available to anyone outside your organization. Keep it safe.
Deploy the public key
Put the public key on every server that uses Pingdom server monitor. The key goes in the Pingdom server monitor config directory: /var/lib/scoutd/scout_rsa.pub
and should be owned by the scoutd
user. E.g. after uploading the key via SSH:
sudo mv scout_rsa.pub /var/lib/scoutd/scout_rsa.pub
sudo chown scoutd:scoutd /var/lib/scoutd/scout_rsa.pub
sudo scoutctl restart
Placement of the public key on new servers should become part of your provisioning process.
Add the Plugin and sign the code
After clicking the "Add Plugin" button you'll see the plugin directory. In the right column under the "Private Plugins" header, click the "Add" button. Fill in the details for your plugin.
After creating the plugin, you'll be prompted to sign the plugin code. You can run the command from any computer with the Pingdom server monitor private key installed (~/.scout/scout_rsa
). The command looks like this:
scout sign http://server.pingdom.com/UNIQUE_PRIVATE_PLUGIN_ID
You're now ready to install the plugin!
Updating Private Plugins
When you change the plugin code, you'll be prompted to run the scout sign command again to update the signature.
Public Plugins
If you've created a plugin that's useful to others, we'd love to add it to our plugin directory. Plugins listed in our public directory are available to everyone and can be installed without placing a file on the server.
To submit your plugin to the public directory, email the url of the private plugin you’d like to share to support.server@pingdom.com. We'll review it for inclusion in the directory.
Alerts are generated by the plugin code itself. Triggers are configured on server.Pingdom.com, and detect changes and thresholds in numerical data reported by plugins. So if you want to generate an alert on a numerical threshold or trend, use Triggers instead of alerts.
Triggers are specified through the web interface at server.pingdom.com and they have a lot of flexibility.
When a trigger fires, an alert is generated for the plugin. There are three kinds of triggers:
- Peak
- Plateau
- Trend
You can configure triggers by clicking on the 'Triggers' menu option when editing a plugin. Note that data must be reported before configuring triggers.
Don't use require or load directly. Instead, use needs, as in:
class MyPlugin < Scout::Plugin
needs 'mysql2'
...
The Scout::Plugin
base class defines needs so missing libraries don't break the checkin process.
Note that Rubygems will be loaded if needed to find the libraries you need -- you don't have to explicitly specify Rubygems.
PSM will catch runtime exceptions thrown inside your plugin, and create an Error informing the user of the issue. A rule of thumb: don't catch any exceptions yourself unless it's something the end user can address, i.e., by changing option values. When that's the case, catch the exception and call error("your informative error message")
to guide the user.
Each plugin can send a maximum of 20 unique metrics to Pingdom server monitor. If more than 20 metrics are used, only the first 20 metrics will be tracked. A new metric can't be substituted for one that is no longer used.
What pragmatic plugin developer wouldn't want unit tests? It's easy to build tests for your Pingdom server monitor plugin.
The Test file
Create a file named test.rb
in the same directory as the Scout plugin file.
Here's an example test file used to test PSM's HAProxy Plugin:
require File.dirname(__FILE__)+"/../test_helper"
require File.dirname(__FILE__)+"/haproxy_stats"
class HaProxyTestTest < Test::Unit::TestCase
def test_truth
assert true # replace me
end
end
plugin=HaproxyStats.new(last_run,memory = {},options = {})
last_run
is a Time
object that indicates the last time the plugin ran. Set this to nil
to simulate the first run of the plugin. This time is accessible in the plugin as @last_run
.
memory
is a Hash
that contains any memory settings for the plugin. For example, you might store the value of a metric in a previous run of a plugin and calculate how the metric has changed in the next run. Learn how to access memory in a plugin.
options
is a Hash
that contains any options for the plugin. For example, you might need an option to specify the port used to access an Apache status page. Learn how to access options in a plugin.
To run a plugin, call the Plugin#run method
:
# initialize
plugin=HaproxyStats.new(nil) # no last run, no memory, and no options
# run!
result = plugin.run
Plugin#run returns a Hash. Here's a sample:
{# when the plugin ran, it stored this in memory for access in the next run.
:memory=>{:stored_disk_usage => 30},
# An Array of report Hashes
:reports=>[{:disk_size=>38.0, :memory_usage=>'691 KB'}],
# An Array of Alerts, each is a Hash
:alerts=>[{:subject=>"Memory Usage Alert",
:body=>"Memory usage has exceeded 95%. Memory: 691KB."}],
# Similar to Alerts
:errors=>[]
}
plugin = LoadAverages.new(Time.now-5*60,{:previous_load => 1.0},
{:num_processors => 1})
result = plugin.run
assert_equal 2.0 result[:reports].first[:cpu_last_minute]
assert_equal 2.0 result[:memory][:previous_load]
assert_equal 'Load Increased', result[:alerts].first[:subject]
$ ruby test.rb
Loaded suite test
Started
.....
Finished in 0.492469 seconds.
5 tests, 14 assertions, 0 failures, 0 errors
Advanced examples
Take a look at the following tests for Scout Plugins available on Github:
- HAProxy - uses FakeWeb to simulate HTTP calls.
- Apache Analyzer - uses Timecop to simulate running a plugin at specific times for log file parsing.
- Overview with Alerts - uses Mocha to stub system calls.
alert('subject')
Or if you've got more to say, include an alert body:
alert('subject','body')
Know when to use an alert or trigger: see Alerts vs triggers above.
Errors
You can also create error messages. You should use errors when something has prevented your plugin from operating properly and the user can do something about it. A good example of an error is a missing gem dependency.
Generate an error with:
error('your error text')
Errors can also have a subject and body:
error('error subject','your error text')
Know when to use an error or exception: Exceptions