README

                Splunk-friendly scripts for monitoring RabbitMQ

                                 Jerry Kuch
                              RabbitMQ/VMware


* INTRODUCTION

This is a collection of Python scripts that use the RabbitMQ
management HTTP API to:

   1. Perform basic health checks of a RabbitMQ broker
   2. Monitor the behavior of your RabbitMQ_based messaging
      application (after you gather a bit of information about your
      application's structure and function)

The README in front of you documents what's in this package, and how
to wire it into your Splunk installation.  The steps here have been
tested with Splunk 4.3, build 115073, running on Ubuntu 11.


* HOOKING RABBITMQ LOGS UP TO SPLUNK

** What RabbitMQ logs to watch

Any RabbitMQ broker has a few logs of interest that you'll want to
watch, in particular:

   - the main broker log (named something like [email protected],
     depending on your system)

   - the Rabbit SASL (Erlang/OTP System Application Support Libraries)
     log (named something like [email protected])

The above logs, although in a slightly idiosyncratic format, are well
parsed by Splunk.  Pretend they're regular syslogs and Splunk is smart
enough to catch the timestamps and whatnot.

You may also want to watch the startup_err and startup_log files,
although you'll typically only look at these when starting or
restarting a broker, or when one fails to start up.  Their format is
not especially Splunk friendly, as they're just captures of Rabbit's
console output, but pulling them into Splunk so they're available at
your regular monitoring dashboard isn't a bad idea.

Note that log locations vary by RabbitMQ version and platform.  For a
quick reference to where they can be found, and what they're called on
your specific platform see:  http://www.rabbitmq.com/configure.html


** Hooking Rabbit logs up to Splunk

The following section assumes you're running RabbitMQ on one of the
popular Linux distributions with respect to where the logs are found.
If you're on a different platform, amend paths as needed based on
http://www.rabbitmq.com/configure.html

To make Splunk aware of the Rabbit logs, go to the Splunk web UI:

   - 'Home' (by clicking on the Splunk logo at the top of the page)

   - click on the 'Add data' button

   - Under 'Choose a Data Type' click "A file or directory of files"

   - Choose "Consume any file on this Splunk server" (or "Forward data
     from files or directories to this Splunk server from another
     server" if that is your destiny and you're using Splunk's
     universal forwarder with remote machines)

   - When prompted, enter the path to the log of your choice,
     e.g. '/var/log/[email protected]'  You can start a new
     sourcetype, giving it a name like 'rabbitmq-logfile' if you'd
     like.  When shown a preview make sure the timestamps are being
     picked out correctly and that breaks between events look correct
     (Splunk does a pretty good job of this by default).

Now go to the search UI and make sure you can see the event sources.


* HOOKING THE RABBITMQ MONITORING SCRIPTS UP TO SPLUNK

** Add a RabbitMQ user to your broker/cluster with 'Monitoring' abilities

   Using the RabbitMQ management web UI;

     - go to the 'Users' tab
     - expand 'Add / update a user'
     - enter a username and password
     - click on the 'Monitoring tag'
     - once the user is added, click on it and set permissions for the
       virtual host you wish to monitor

   At this point you may want to perform a quick test by invoking the
   script api_ping_check.py manually as follows:

     ./api_ping_check.py rabbitmqhost:55672 vhostname username password

  If you get something like:

     2012-02-23 17:24:09.422964 OK RabbitMQ API ping check, broker
     localhost:55672 alive: {"status":"ok"}

  All is well.  If you get a 401, you probably got the
  username/password wrong, or else forgot to grant appropriate virtual
  host permissions to the user you created above.

** The wrapper script

   - TODO:  why does wrapper exist, with reference
   - TODO:  how does wrapper work

** Connecting wrapped scripts to Splunk

   - TODO:  walk through how to do this in Splunk UI

** The monitoring scripts in this package

   - TODO:  list the scripts and their usages

* REFERENCES

  Splunk installation manual:
  http://docs.splunk.com/Documentation/Splunk/latest/Installation/Startsplunkforthefirsttime

  Starting Splunk for the first time:
  http://docs.splunk.com/Documentation/Splunk/latest/Installation/Startsplunkforthefirsttime
    Covers:
      - starting the Splunk daemon
      - accessing the Splunk web UI

  Adding new Python to the Splunk environments:
  http://splunk-base.splunk.com/answers/8/can-i-add-python-modules-to-the-splunk-environment