roll no rocks

Extending logwatch

Using logwatch to report on unattended-upgrade’s “restart required”

At TRI Resources International, we automate as much of our of our systems management work as possible.  A couple of the tools we use with our linux servers are logwatch to keep us up to date on server health, and the unattended-upgrades package to automatically install security updates.

Problem is, the basic logwatch report doesn’t include any notice when a package update requires a system reboot. So we customized it. Here’s how to add unattended-upgrade “restart required” info to logwatch reports.

Logwatch overview

It turns out logwatch is pretty simple.  It just reads and filters log files and creates a report including the matching entries.  Each report section in logwatch is called a “service,” and a service is defined by three parts:

  • A logfile group configuration file indicates which log files the service will use, and can optionally filter the lines.
  • A service configuration file indicates which logfile group to use and may include a title.
  • A script file simply takes whatever log file lines it’s given through stdin, and prints out the report content.

The logwatch docs are here.  After digesting how it works, they actually seem to be rather clear and complete.  The only thing missing is a few examples (so… this post.)  Logwatch does all the plumbing so most of the setup for creating a new service is just configuration.

Creating the “restart” logwatch service

When the unattended-upgrades package installs an upgrade requiring a restart, two files are created:

  • /var/run/reboot-required is created with a single line warning message
  • /var/run/reboot-required.pkgs is created listing each of the packages that require a restart

While these are not exactly log files, we can use them for input to a logwatch service.  The idea is to create a service to scan these files, and when they are not empty, add a section to the logwatch report entitled “Restart required” and including the contents of the files.

Since we’re using non-standard files a log input, we’ll need to create a custom logfile group.  The logfile group configuration file is:

# File: /etc/logwatch/conf/logfiles/restart_logs.conf

LogFile = /var/run/reboot-required
LogFile = /var/run/reboot-required.pkgs

*Remove = ^\s*$

Lines 3 and 4 just indicate which files to use as input.  Line 6 uses the “Remove” filter to drop blank lines.  The built-in filters are described well in the docs. Most of them have to do with filtering different date formats or other content, but the Remove filter is “Just a case insensitive, inverse egrep.”  The only unexpected issue was that the argument must be a raw regular expression with no enclosing quotes or other delimiters.  I could have skipped using a filter here and cleaned up the results later in the script file, but it’s less code to do it this way.

The service configuration file is just:

# File: /etc/logwatch/conf/services/restart.conf

Title = "Restart required"
LogFile = restart_log

Line 3 creates a title that will be used for this section in the logwatch reports, and line 4 indicates that the input will be from the logfile group created above.

Finally the script file. This is the only one with any real code, and not much.

#!/usr/bin/perl
# File: /etc/logwatch/scripts/services/restart

while (defined($ThisLine = )) {
   print $ThisLine;
}
exit(0);

We’re not grouping or filtering anything here, or deciding what to display based on the logwatch detail level, so this just reads each line from stdin and prints it to the output. The docs suggest that a one-line bash script (“cat“) would do the same thing, but it seemed appropriate to write this in perl to match the rest of the code.

Note that while the service config file must specify the logfile group explicitly, the script file is associated with the service by having the same file name. So in this case the restart.conf service config file expects a script file named restart with no file extension.

 

That’s it!  Really.  When a reboot’s not required, the /var/run files are missing and the report section is not created.  When an upgrade requires a reboot, you’ll get a section in the logwatch report like this:

--------------------- Restart required Begin ------------------------

*** System restart required *** 
linux-image-2.6.35-25-generic_2.6.35-25.44_amd64

---------------------- Restart required End -------------------------

 

 

Tom

,