Help Docs ~ Server Monititor

Installation

An easy-to-install, lightweight daemon

Scoutd overview

If you are using an existing version of our agent (the PSM Ruby gem), see Upgrading to scoutd. If you have a fresh server, see Installing scoutd.

Installing scoutd

If you are using our older Pingdom Server Monitor agent that is run via Cron, see Upgrading to scoutd. These instructions are for a fresh server w/o the older version of our agent installed.

Installing scoutd automatically using the installer:

$ curl -Sso scout_install.sh https://server.pingdom.com/scout_install.sh && sudo /bin/bash ./scout_install.sh -k <YOUR_PSM_KEY>

We recommend using our installer to install scoutd. If you prefer to install scoutd manually, refer to the section for Installing scoutd manually.

Updating scoutd to the latest version

If you have already installed scoutd and would like to make sure you have the latest version

  • Ubuntu/Debian
sudo apt-get update && sudo apt-get install scoutd
  • RedHat/CentOS/Fedora/Amazon/Scientific/Oracle
sudo yum install scoutd

Configuration

  • scoutd is configured via a YAML config file, /etc/scout/scoutd.yml.
  • The daemon runs under the scoutd user. If you have plugins that read files and/or run commands, ensure scoutd has appropriate permissions. Refer to the scoutd permissions section.
  • Custom agent files (local plugins, account key, etc) are located in /var/lib/scoutd.

Configuration file options for /etc/scout/scoutd.yml

Config Option Name
Default Value
RubyGem Equivalent

account_key:

hostname:

The short hostname of the server

--hostname

display_name:

-n, --name

log_file:

/var/log/scout/scoutd.log

ruby_path:

First 'ruby' found in $PATH

environment:

-e, --environment

roles:

-r, --roles

agent_data_file:

/var/lib/scoutd/client_history.yml

-d, --data

http_proxy:

--http_proxy

https_proxy:

--https_proxy

log_level:

-l

statsd:
  enabled:
  addr:

 
true
127.0.0.1:8125

Example configuration file

account_key: YOUR_PSM_KEY
ruby_path: /usr/local/rvm/wrappers/ruby-2.1.4/ruby
environment: production
roles: app,nginx

Generating a scoutd configuration file using scoutd

You can use scoutd to generate a scoutd.yml configuration file. Just specify the command line options you want to set in the configuration file followed by the “config” subcommand:

$ scoutd --key=YOUR_PSM_KEY --environment=production --roles=app,nginx config
# generated by 'scoutd config'
account_key: YOUR_PSM_KEY
environment: production
roles: app,nginx

You can copy/paste the output into /etc/scout/scoutd.yml

You can also have scoutd write the output directly to the configuration file by adding the -o option after the config subcommand:

$ scoutd --key=YOUR_PSM_KEY --environment=production --roles=app,nginx config -o

General configuration defaults

  • The scoutd package creates the user and group ‘scoutd’ with a home directory /var/lib/scoutd.
  • The default configuration file is /etc/scout/scoutd.yml. This file is in YAML format and is automatically generated by the installer or when provide your scout key in the YOUR_PSM_KEY variable during manual scoutd package installation.
  • The default log file is /var/log/scout/scoutd.log.

Restart scoutd after making configuration changes

sudo scoutctl restart

Where to place local plugins, RSA key, plugin properties template file, etc

Your custom plugins, RSA key (scout_rsa.pub), plugins.properties, should be placed in /var/lib/scoutd:

$ ls -1 /var/lib/scoutd
37233021.rb
plugins.properties
scout_rsa.pub
task_server_check.rb

These files should be owned by user/group scoutd/scoutd.

Troubleshooting

Displaying scoutd command options

scoutd --help

To display subcommand options, specify the subcommand before the --help flag:

scoutd config --help

Position of arguments when running scoutd

scoutd has both application options and subcommand options. You must specify the application options before the subcommand. You should specify subcommand options after the subcommand:

scoutd [application OPTIONS] config [config-OPTIONS]

For example:

scoutd -k YOUR_PSM_KEY -e prod -r database,mysql config -o

Why am I getting “unknown flag” errors when I try to specify command line options to scoutd?

The position of the command line arguments matters. Refer to Position of arguments when running scoutd

Where to look for logs

/var/log/scout/scoutd.log
/var/log/upstart/scout.log (Ubuntu)
/var/lib/scoutd/latest_run.log
/var/lib/scoutd/scout_streamer.log (realtime)

Enable verbose logging

To enable debug logging, edit /etc/scout/scoutd.yml and set log_level: debug. Restart scoutd with scoutctl restart

How to dump troubleshooting info

When the scoutd service is up and running, send a USR1 signal to have scoutd dump troubleshooting info into the log /var/log/scout/scoutd.log:

sudo killall -USR1 scoutd

How to manually run scoutd and have it log to STDOUT

sudo su - scoutd -c '/usr/bin/scoutd --logfile=- start'

Using scoutd with RVM

See Specifying an alternate Ruby.

YAML formating for /etc/scout/scoutd.yml

If you see the message

scoutd: 2014/11/13 12:49:15 Could not open config file: readfile("/etc/scout/scoutd.yml"): cannot append scalar to non-scalar node

This likely means your configuration file is not in YAML format. Check to make sure the configuration options are in the correct format:

option: setting

This is incorrect:

option=setting
option = setting

Known Bugs

  • Realtime does not currently work with ruby 1.8
  • Realtime doesn't support proxies

Granting permissions to the scoutd user/group

The daemon now runs under the scoutd user and scoutd group. If you were previously running the Ruby agent as root, you may need to give scoutd permission to read log files or run sensitive commands in order for certain plugins to work.

For example, the MySQL Slow Queries plugin requires access to the Mysql slow query log file at /var/log/mysql/mysql-slow.log. This file is usually restricted to the mysql user and adm group on Ubuntu systems. To allow scoutd to read the file, confirm the adm group can read the file and add the scoutd user to the adm group:

# Allow group read
$ sudo chmod -R g+sr /var/log/mysql/

# Find the group name of the slow query log file
$ sudo stat -c%G /var/log/mysql/mysql-slow.log
adm

# Add `scoutd` user to the group from the previous step
$ sudo usermod -a -G adm scoutd

# Restart scoutd so it can pick up the new group permission:
$ sudo scoutctl restart

Executing commands as root via sudo

If your plugin needs to run commands as root, you may need to configure sudo access. For example, the Passenger Plugin must be run as root.

Note: always use the visudo command to edit your sudoers file!

To explicitly list each command that scoutd can run as root, run visudo and use a Cmnd_Alias line to list each command, separated by commas. E.g:

Cmnd_Alias SCOUTD_CMDS = /usr/bin/passenger-status, /usr/bin/passenger-memory-status
scoutd ALL=(ALL) NOPASSWD:SCOUTD_CMDS

If you don't want to list every command you can skip the Cmnd_Alias and simply use NOPASSWD:ALL:

scoutd ALL=(ALL) NOPASSWD:ALL

Note: The requiretty option must be disabled for the scoutd user. If your sudoers file contains Defaults: requiretty, you can either comment it out or specifically disable it for the scoutd user:

# Commented out
# Defaults       requiretty

# Do not enforce requiretty for the scoutd user
Defaults:scoutd       !requiretty

Specifying an alternate ruby

If you'd like scoutd to use a different install of Ruby than the first Ruby found in your $PATH environment variable, set the ruby_path option in the scoutd config file.

When to use ruby_path?

  • You don't have a system Ruby installed (Ruby was installed via RVM)
  • You have plugins that have gem dependencies and those gems are not installed in your system Ruby.
  • You are using Bundler.

Updating your `ruby_path`

1. Add the path to /etc/scout/scoutd.yml

Open the config file:

sudo vi /etc/scout/scoutd.yml

Add the ruby_path to your desired Ruby:

account_key: YOUR_PSM_KEY
ruby_path: /usr/local/rvm/wrappers/ruby-2.1.4/ruby

NOTE: Make sure you use the path to the correct WRAPPER script when using RVM.

2. Restart scoutd

$ sudo scoutctl restart

3. Verify the Ruby path in the scoutd log

$ sudo grep "Found Ruby at path" /var/log/scout/scoutd.log
scoutd: 2014/11/13 12:05:32 Found Ruby at path: /usr/local/rvm/wrappers/ruby-2.1.4/ruby

Using bundler

1. Create a bundler wrapper script

sudo vi /var/lib/scoutd/bundler_wrapper.sh

The wrapper script should cd to the directory where you have your bundle installed and invoke the bundler binary, passing all arguments through:

#!/bin/sh
cd /srv/site/project && /usr/local/rvm/wrappers/ruby-2.1.4/bundle exec $*

2. Set the ruby_path in /etc/scout/scoutd.yml to point to the bundle wrapper script.

Open the config file:

sudo vi /etc/scout/scoutd.yml

Set the ruby_path to the bundler_wrapper.sh script:

account_key: YOUR_KEY
ruby_path: /var/lib/scoutd/bundler_wrapper.sh

3. Restart scoutd

sudo scoutctl restart

4. Verify the Ruby path in the scoutd log

$ sudo grep "Found Ruby at path" /var/log/scout/scoutd.log
scoutd: 2014/11/18 15:50:05 Found Ruby at path: /var/lib/scoutd/bundle_wrapper.sh

Using rbenv

1. Create an rbenv wrapper script

sudo vi /var/lib/scoutd/rbenv_wrapper.sh

The rbenv wrapper script should set RBENV_ROOT which points to your rbenv installation path:

#!/usr/bin/env bash

# Point RBENV_ROOT to where you have rbenv installed.
export RBENV_ROOT="/home/vagrant/.rbenv"

# Note: the parent directory of rbenv must allow the scoutd user
# permission to access the .rbenv directory. Refer to the 
# "Granting permissions to the scoutd user/group" section for
# more details. Example:
#
#  ls -al /home/
#    drwx------  8 vagrant  vagrant  4096 Sep  9 22:11 vagrant
#
# sudo usermod -a -G vagrant scoutd
# sudo chmod g+x /home/vagrant

# Optionally, specify a version to use
# export RBENV_VERSION="2.2.3"

# Exec rbenv
$RBENV_ROOT/bin/rbenv exec ruby $*

2. Allow the script to be executed and set the ruby_path in /etc/scout/scoutd.yml to point to the bundle wrapper script.

Set execute permissions:

sudo chown scoutd:scoutd /var/lib/scoutd/rbenv_wrapper.sh
sudo chmod 750 /var/lib/scoutd/rbenv_wrapper.sh

Open the scoutd.yml config file:

sudo vi /etc/scout/scoutd.yml

Set the ruby_path to the bundler_wrapper.sh script:

account_key: YOUR_KEY
ruby_path: /var/lib/scoutd/rbenv_wrapper.sh

3. Restart scoutd

sudo scoutctl restart

4. Verify the Ruby path in the scoutd log

$ sudo grep "Found Ruby at path" /var/log/scout/scoutd.log
scoutd: 2014/11/18 15:50:05 Found Ruby at path: /var/lib/scoutd/rbenv_wrapper.sh

Using scoutd with HTTP proxies

Pingdom server monitor communicates via HTTP and HTTPS. All communication is outgoing, so no incoming ports need to be open.

To use proxies, set the http_proxy and https_proxy options in the /etc/scout/scoutd.yml configuration file:

http_proxy: http://proxy.io
https_proxy: https://proxy.io

Note: Realtime does not yet work when using proxies.

Scoutd requirements

  • Supported distros:
    • Ubuntu 10.04, 12.04, 14.04
    • Debian Jessie/8, Wheezy/7
    • RedHat 6/7. Distros based on RedHat are also supported:
      • CentOS
      • Amazon Linux AMI
      • Oracle
      • ScientificLinux
    • Fedora 20+
  • Ruby 1.8.7 or higher see Specifying an alternate Ruby. If you're upgrading from the old agent, you have Ruby already.

Reverting to the old Pingdom server monitor agent

If you are having issues upgrading to scoutd, you can revert back to the old PSM agent.

1. Stop scoutd, remove the scoutd package

$ sudo scoutctl stop

2. Install the scout gem

$ sudo gem install scout

3. Re-enable the Cronjob for Scout

* * * * * root /usr/local/bin/scout SyX4mVBFkLkMwwYgqFMIUZ9EWHCFnThZQK -e prod -r app,nginx

4. Remove the scoutd package

  • Ubuntu
$ sudo dpkg --purge scoutd
  • RedHat, CentOS, Amazon, Fedora
$ sudo yum remove scoutd

Upgrading to scoutd

scoutd is a replacement for your existing scout gem. scoutd is a binary and doesn't add any dependencies to your existing setup.

What changes with scoutd?

  • scoutd is a daemon while your existing scout agent runs via a Cron job.
  • scoutd is configured via a config file while the scout agent's configuration is specified in the Cron job.
  • The daemon runs under the scoutd user. If you have plugins that read files and/or run commands, ensure scoutd has appropriate permissions. Refer to the scoutd permissions section.
  • File locations are now standardized. The daemon configuration file is at /etc/scout/scoutd.yml and custom agent files (local plugins, account key, etc) are now located in /var/lib/scoutd. The installer will copy your existing custom agent files for you.

Upgrading automatically using the installer:

$ curl -Sso scout_install.sh https://server.pingdom.com/scout_install.sh && sudo /bin/bash ./scout_install.sh

We recommend using our installer to upgrade from the cron agent to scoutd. If you prefer to upgrade to scoutd manually, refer to the section for Upgrading scoutd manually.

Upgrading scoutd manually

1. Install the scoutd package

  • Ubuntu
wget -q -O - https://archive.scoutapp.com/scout-archive.key | sudo apt-key add -
echo "deb http://archive.scoutapp.com `lsb_release -s -c` main" | sudo tee /etc/apt/sources.list.d/scout.list
sudo apt-get update
sudo env SCOUT_KEY=YOUR_PSM_KEY apt-get install scoutd

Replace YOUR_PSM_KEY with your Pingdom server monitor account key.

  • RedHat, CentOS
sudo wget -O /etc/yum.repos.d/scout.repo https://archive.scoutapp.com/yum.repos.d/scout-rhel.repo
sudo rpm --import https://archive.scoutapp.com/RPM-GPG-KEY-scout
sudo env SCOUT_KEY=YOUR_PSM_KEY yum install scoutd

Replace YOUR_PSM_KEY with your Pingdom server monitor account key.

  • Amazon Linux
sudo wget -O /etc/yum.repos.d/scout.repo https://archive.scoutapp.com/yum.repos.d/scout-amazon.repo
sudo rpm --import https://archive.scoutapp.com/RPM-GPG-KEY-scout
sudo env SCOUT_KEY=YOUR_SCOUT_KEY yum install scoutd

Replace YOUR_PSM_KEY with your Pingdom server monitor account key.

  • Fedora
sudo wget -O /etc/yum.repos.d/scout.repo https://archive.scoutapp.com/yum.repos.d/scout-fedora.repo
sudo rpm --import https://archive.scoutapp.com/RPM-GPG-KEY-scout
sudo env SCOUT_KEY=YOUR_SCOUT_KEY yum install scoutd

Replace YOUR_PSM_KEY with your Pingdom server monitor account key.

2. Disable the Pingdom server monitor Cron job

Comment out the line that looks like this:

* * * * * root /usr/local/bin/scout YOUR_PSM_KEY -e prod -r app,nginx

Your Cronjob is either in the system-wide Cron file (vi /etc/crontab) or in the user that runs PSM's Cron file (crontab -e).

3. Configure scoutd

Pass the same flags used in your Cron job to scoutd [options] config. For example, the flags in the Cronjob above are -e prod -r app,nginx, so we'll pass through:

sudo scoutd --key=YOUR_PSM_KEY -e prod -r app,nginx config -o

4. Copy your PSM config directory

Copy the contents of the .scout directory to /var/lib/scoutd/ and set the owner to scoutd:

$ sudo cp -r /home/USER_RUNNING_SCOUT/.scout/* /var/lib/scoutd/
$ sudo chown -R scoutd:scoutd /var/lib/scoutd

5. Restart scoutd

$ sudo scoutctl restart

$ sudo tail /var/log/scout/scoutd.log
scoutd: 2014/11/14 17:22:36 Using Configuration:
...
scoutd: 2014/11/14 17:23:42 Agent available

6. Uninstall the scout gem

$ gem uninstall scout

Installing scoutd manually

If you are using our older PSM agent that is run via Cron, see Upgrading to scoutd. These manual installation instructions are for a fresh server w/o the older version of our agent installed.

1. Verify Ruby is installed

The scoutd daemon runs Ruby plugin plugins and requires Ruby 1.8.7+. To verify that Ruby is installed:

$ ruby -v
ruby 2.1.2p95 (2014-05-08 revision 45877) [x86_64-linux]

See installing Ruby if Ruby isn't installed.

2. Install the scoutd package

  • Ubuntu
wget -q -O - https://archive.scoutapp.com/scout-archive.key | sudo apt-key add -
echo "deb http://archive.scoutapp.com `lsb_release -s -c` main" | sudo tee /etc/apt/sources.list.d/scout.list
sudo apt-get update
sudo env SCOUT_KEY=YOUR_PSM_KEY apt-get install scoutd

Replace YOUR_PSM_KEY with your Pingdom server monitor account key.

  • RedHat, CentOS
sudo wget -O /etc/yum.repos.d/scout.repo https://archive.scoutapp.com/yum.repos.d/scout-rhel.repo
sudo rpm --import https://archive.scoutapp.com/RPM-GPG-KEY-scout
sudo env SCOUT_KEY=YOUR_PSM_KEY yum install scoutd

Replace YOUR_PSM_KEY with your Pingdom server monitor account key.

  • Amazon Linux
sudo wget -O /etc/yum.repos.d/scout.repo https://archive.scoutapp.com/yum.repos.d/scout-amazon.repo
sudo rpm --import https://archive.scoutapp.com/RPM-GPG-KEY-scout
sudo env SCOUT_KEY=YOUR_PSM_KEY yum install scoutd

Replace YOUR_PSM_KEY with your Pingdom server monitor account key.

  • Fedora
sudo wget -O /etc/yum.repos.d/scout.repo https://archive.scoutapp.com/yum.repos.d/scout-fedora.repo
sudo rpm --import https://archive.scoutapp.com/RPM-GPG-KEY-scout
sudo env SCOUT_KEY=YOUR_PSM_KEY yum install scoutd

Replace YOUR_PSM_KEY with your Pingdom server monitor account key.

3. Specifying roles, environment, etc. (optional)

If you require additional configuration options, see Configuration.

Uninstalling scoutd

  • Ubuntu, Debian
sudo dpkg --purge scoutd
  • RedHat, CentOS, Amazon, Fedora
sudo yum remove scoutd

Installation

An easy-to-install, lightweight daemon