Application Monitoring

Resque

Resque has a built-in web UI to look at jobs mounted at /jobs. This is also linked under Manage → Manage Worker Jobs in the main navigation. This is only available to users in the administrators group.

System status

The Avalon web application provides a status monitoring page at /about/health. It reports on the current status of major components:

Fedora
Solr
Database (MySQL, Oracle, etc.)
Mediainfo
Streaming Server
Redis

The HTML view of the page simply provides a list of services and a badge reading OK or Error (along with any service-provided error messages), and a link to click through for a detailed view of each service.

Machine-readable status

By appending .xml, .json, or .yaml to the URL (e.g., /about/health.json), you can retrieve a machine-readable status report that can be parsed and acted on by any number of enterprise monitoring tools.

The main /about page (also retrievable as .xml, .json, or .yaml) provides much more information about the system, including Fedora/Solr/Database configuration info, a full list of Matterhorn services and status, and a complete list of installed Ruby gems (and versions) required by Avalon.

To retrieve the same report from the command line instead of via HTTP:

$ cd /var/www/avalon/current
 
# Concise service health/status report:
$ bundle exec rake about:health[format]
# Full configuration report:
$ bundle exec rake about:report[format]
# (where format is xml, json, or yaml)

This page is behind authentication; a privileged user must be logged into the Avalon instance for status monitoring files and web pages to be accessible.