Centralized Syslog Using MongoDB And PHP

Introduction

This setup codenamed „PMS“ (for PHP/MongoDB/Syslog) can be used to let one or more Syslog-servers write log messages to a MongoDB collection. They can then be accessed using a PHP web application.

pms architecture diagram.

It has been tested on Linux using RSyslog, MongoDB, Apache and PHP (with MongoDB extension) from Devuan (Ceres).

The quality of this software is currently alpha, as development is stabilizing. The user interface is not yet consistent.

Attention: pms is currently in „alpha“ state.

Only 1 filter field (field1) is supported.
Browsing by 25 records back/forward is currently not consistent when changing to the previous hour.

I recommend using this software only if you are willing to get involved with this work in progress.

The setup instructions for MongoDB and RSyslog have been verified and are usable.

System Requirements

RSyslog with module „ommongodb“ enabled, either

apt-get install rsyslog rsyslog-mongodb

or, if your distribution does not offer a packaged MongoDB output module for rsyslog, compile and install rsyslog manually as explained in [RSYSLOG-MONGODB].

MongoDB server.
Webserver that runs PHP scripts.
PHP with extension „mongodb“ described in [PHP-MONGODB].

Preparations

Prepare a MongoDB database „syslog“ with a collection „log“:

use syslog
db.createCollection('log')

Alternatively, create a „capped collection“, a feature that can be very useful for temporary, analytical storage of log messages. The capping feature will expire the oldest records when a maximum of allowed records is reached, effectively turning the collection into a large ring-buffer.

Older records get expired when a defined limit is reached.

In this example, the collection will be capped as soon as either 1 million records or 1 GB of storage are reached:

db.createCollection('log', { capped : true, size : 1000000000, max : 1000000 } )

To access the database with external client programs, add a user „john“ for this database:

db.addUser('john', 'secret')

Redirect rsyslog into MongoDB by adding a configuration file /etc/rsyslog.d/49-ship-syslog.conf:

module (load="ommongodb")
*.* action(type="ommongodb"
           server="localhost" port="27017" db="syslog" collection="log"
           uid="john" pwd="secret")

This configures rsyslog to write all log messages to a MongoDB database „syslog“, collection „log“ using the database user „john“ created previously.

Verification of Syslog Data

Using a MongoDB desktop client (example: [ROBOMONGO]), you can verify that datasets appear in the collection:

Screenshot of robomongo browsing the syslog database, showing that documents are arriving in the log collection.

PMS Installation

Download the software:

wget http://tk-sls.de/ref/pms.tar.gz
tar xzf pms.tar.gz

or get the latest development code with Subversion (ViewVC):

svn co svn://tk-sls.de/pms

Move the folder „pms“ to a subdirectory of your webserver’s document root, adjusting permissions to the webserver account:

mv pms /var/www/pms
chown -R www-data:www-data /var/www/pms

Configuration

Edit the file /var/www/pms/include/config.inc.php and set MongoDB database host and port as well as the database user credentials you set earlier:

<?php
$config = array(
 'user' => 'john',
 'password' => 'secret',
 'host' => 'localhost',
 'port' => '27017',
 'db' => 'syslog',
 'collection' => 'log',
  ...

Usage

The software offers a simple timeline-paginated browser through a stream of log messages. Navigating through logs is done by moving forward and backward in the log messages by either 1 hour time intervals or by 25, 50 or 100 messages per page.

pms-1.0.0 syslog browser.

The time interval can be changed to „minute“, „hour“ or „day“, and the output can be paginated by multiples of those (for example „15 minutes“ or „12 hours“). Edit the settings time_delta and time_unit in config.inc.php:

<?php
$config = array(
  ...
 'time_delta' => 1,
 'time_unit' => 'hour',
  ..