Part Two: Configuring a Central Syslog Server

By Urbana Der Ga’had

In Part 1 of this article we installed and configured LogSentry to automate the monitoring of the system logs on our machine. That’s great if we’ve only got one or two systems, but what if we have a dozen *nix systems, or fifty, or one hundred? I can tell you right now we don’t want to maintain one hundred separate sets of logcheck.ignore! Even if we used a single master file, we’d still have to distribute it to all systems, and maintain the LogSentry software itself. This is a job we should centralize.

In addition to the obvious advantages of centralizing the log monitoring, sending the system log entries to a centralized syslog server has security benefits. If your machine gets hacked, the intruder will most likely wipe all traces of the intrusion from the logs. At least with a central syslog server you’ll get the entries associated with the attack itself, even if the intruder trojans syslog later and you don’t get any more meaningful messages once the box has been rooted. This can be equally helpful in the case of hardware failure or other disaster.

What follows is a simple centralized syslog setup, suitable for systems that are inside a firewall. In Part Three of this article, we will look at ways to improve security and performance.

Configuring the Server

First consider the requirements and decide what system you’re going to run as syslog server. Consider that parsing the logs will consume a fair amount of memory and CPU, and this requirement will increase as you add machines to your network. The system should be hardened and fully patched. It should not be accessible to users on the network. Think of this machine as an integral part of your network security infrastructure. Also, /var should be its own file system and it should be big enough to hold the log messages from all your machines, keeping in mind whatever rotation scheme you use (so for instance, you might have four sets of logs in /var/log or /var/adm at a given time.) Multiply the number of machines by the size of the system log directory on your busiest machine, like a mail server. That should hold you for a while.

The only special configuration needed on the syslog server is a change to the command options used to start syslog. It needs to be started with the -r option. This allows the syslog facility on your server to receive messages from the network.

On many systems this change will be made directly in the rc script used to start syslogd at boot time. Other systems use configuration files *for* the rc scripts, like Red Hat. No matter what, read the script and you’ll find where the option can be specified. On Red Hat the options are read from /etc/sysconfig/syslog.

So on Red Hat we edit /etc/sysconfig/syslog and change this:
SYSLOGD_OPTIONS="-m 0"
to this:
SYSLOGD_OPTIONS="-m 0 -r"

Then restart syslogd. Install and configure LogSentry. Your server is now ready for action.

Configuring the Clients

The necessary changes for the clients are within the /etc/syslog.conf file. If you’ve got a bunch of systems with vanilla syslog.confs you can just make a new, slightly less vanilla configuration file and distribute it to all of your *nix systems with scp or rsync. For each appropriate entry in your syslog.conf (the ones that are writing to files), add an entry that sends them off to syslog server. Be careful with the file, it is sensitive to white space — you must use tabs between the fields. Here’s an example:

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none  /var/log/messages
*.info;mail.none;authpriv.none;cron.none        @10.10.10.15 

Now you can restart syslogd on the client. If you tail the messages file on syslog server, you should start to see all the lovely messages from the clients. Verify that you are getting your LogSentry emails. Now you can start developing the filters in logcheck.ignore. Omit the host name for patterns that should be ignored on all systems. Keep the host name in the message if it should apply to a specific host. The biggest pain in the butt is going to be developing your log filters. Remember, that which is not explicitly permitted is forbidden!