Managing your Linux/Unix log files using logrotate

Log files are one of the most important files where almost all precious and sometimes unnecessary information are stored in regard to your server’s running state. For example, if your system’s security has been breached or compromised, it’s these log files which will come to your rescue to help you identity where or what went wrong.

In case if you don’t know, your Linux/Unix server is currently logging kernel and security logs in the file called /var/log/messages. Just do a simple ” tail -f /var/log/messages ” to get feel and see the actual current logs generated by various daemons running on your system.

Now if your server also has a Apache Web server or a Squid Proxy server running and you want to manage their respective logs in your own fashion, then the following information might help you out.

 

First of all, you will need the program called “logrotate”. Logrotate is very useful utility which can rotate log files and archive them in a location that you specify. We will be using “logrotate” in conjunction with “cron“.

In Linux/Unix, cron is a time-based scheduling service in Unix-like computer operating systems. It is available on almost all versions of Linux and Unix.

Having said that, logrotate should be installed in your Linux/Unix distribution but if is not, simply use your system package management system to install it.

For example, for Debian based system, all you need to do to install logrotate is:

apt-get install logrotate

For this guide, we will be rotating and managing the log files generated by Apache and Squid on a FreeBSD-6.x and a Linux Debian-4.1 box. However, it should be also work on other Linux distributions like RedHat, Slackware or SuSE since the fundamentals are the same of all Linux based distributions.

I also assume that your Apache logs are kept in /var/log/apache and your Squid logs are kept in /var/log/squid.

On a FreeBSD-6.x box:

(1.) Make and Install from ports:

cd /usr/ports/sysutils/logrotate

(2.) Configure and Compile

make install clean

If all goes well, we are done and logrotate is installed.

(3.) Create a new logrotate.conf file.

vi /usr/local/etc/logrotate.conf

# Added the following to rotate Apache and Squid logs

# see “man logrotate” for details
# rotate log files weekly
#weekly
daily

# keep 4 weeks worth of backlogs
rotate 7

# send errors to root
#errors root

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
compress

# RPM packages drop log rotation information into this directory
include /usr/local/etc/logrotate.d

/var/log/lastlog {
monthly
rotate 12
}

# system-specific logs may be configured here

(4.) Create a directory for specific logrotate files

mkdir -p /usr/local/etc/logrotate.d

(5.) First, create a logrotate file for Squid to rotate it’s access.log files for 90 days and cache.log for 7 days.

cd /usr/local/etc/logrotate.d/

vi /usr/local/etc/logrotate.d/squid

#Copy and paste the following

/var/log/squid/access.log {
daily
rotate 90
copytruncate
compress
notifempty
missingok
}
/var/log/squid/cache.log {
daily
rotate 7
copytruncate
compress
notifempty
missingok
}

(6.) Create the necessary directories and files for logrotate and test and debug logrotate

mkdir /var/lib/

touch /var/lib/logrotate.status

/usr/local/sbin/logrotate -d /usr/local/etc/logrotate.conf
/usr/local/sbin/logrotate -f /usr/local/etc/logrotate.conf

(7.) Next, we will rotate and manage Apache logs

vi /usr/local/etc/logrotate.d/apache

#Add the following to rotate and manage Apache access_log and error_log for 30 days.

#Note: If your Apache logs may be in a different directory, simply change the directory.

/var/log/apache/access_log {
daily
rotate 30
copytruncate
compress
notifempty
missingok
}
/var/log/apache/error_log {
daily
rotate 30
copytruncate
compress
notifempty
missingok
}

If all goes well, that’s it. Your Apache and Squid logs should be rotated.

The last thing is to add an entry into crontab and letting the cron daemon rotate your Apache and Squid logs automatically.

(8.) Automating logrotate using crontab

vi /etc/crontab

#Add the following to rotate your logs at 1 AM in the morning

#Logrotate
0 1 * * * root /usr/local/sbin/logrotate /usr/local/etc/logrotate.conf > /dev/null 2>&1

That’s it. Your Apache and Squid logs will be rotating without manual intervention!!

Using logrotate on a Debian-4.1 box

(1.) Install the logrotate program

apt-get install logrotate

(2.) Create the necessary directories and files

mkdir -p /var/lib/logrotate/

touch /var/lib/logrotate/status

mkdir -p /etc/logrotate.d/

(3.) Create a new logrotate.conf

vi /etc/logrotate.conf

#Copy and paste the following

# see “man logrotate” for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp, or btmp — we’ll rotate them here
/var/log/wtmp {
missingok
monthly
create 0664 root utmp
rotate 1
}

/var/log/btmp {
missingok
monthly
create 0664 root utmp
rotate 1
}

# system-specific logs may be configured here
(4.) Create the squid logrotate file to rotate and manage access.log for 90 days and cache.log for 7 days.

vi /etc/logrotate.d/squid

#Copy and paste the following

/var/log/squid/access.log {
daily
rotate 90
copytruncate
compress
notifempty
missingok
}
/var/log/squid/cache.log {
daily
rotate 7
copytruncate
compress
notifempty
missingok
}

(5.) Create the Apache logrotate file to rotate and manage access_log for 30 days and error_log for 30days.

vi /etc/logrotate.d/apache

#Copy and paste the following. Note: your apache log’s directory might be different. Simply change the path of your directory.

/var/log/apache/access_log {
daily
rotate 30
copytruncate
compress
notifempty
missingok
}
/var/log/apache/error_log {
daily
rotate 30
copytruncate
compress
notifempty
missingok
}
(6.) Test and debug your logrotate configuration for any errors

/usr/sbin/logrotate -d /etc/logrotate.conf

/usr/sbin/logrotate -f /etc/logrotate.conf

If all goes well, you are good to go.

(7.) Now all that is left is to automate the logrotate process from crontab

vi /etc/crontab

#Copy and paste the following

#Logrotate at 1 AM in the morning

0 01 * * * root /usr/sbin/logrotate /etc/logrotate.conf > /dev/null 2>&1

That’s it! The cron daemon will automatically rotate your Apache and Squid logs at 1 AM on a daily basis.

 

 

To show how you can use logrotate for your own applications, let’s look at an example that will come in handy for a lot of people: rotating logs for your custom virtual hosts. We’ll use apache for this example, but it can be tweaked pretty easily for other web servers like nginx or lighttpd, usually just by changing the postrotate script.

First we’ll want to create a file to hold the configuration that will tell logrotate what to do with the virtual host’s log files. We won’t edit the main config file or the web server’s config file, since there’s always a possibility that a future package upgrade might want to overwrite the config. Instead we’ll make our own. Let’s call it:

/etc/logrotate.d/virtualhosts

This example tosses all the virtual hosts into one file, but if you have one that’s busier than others you may want to create separate config files to handle the needs of your different domains. We’ll also specify several items that are probably already set in your main config, just so we cover all the bases.

The files

We’ll say that we have two virtual domains, domain1.com and domain2.com, and that the log files for each are in /home/demo/public_html/(domain name)/log. The first thing we’ll do in our config file is tell logrotate where to find the log files, then start the config block for them:

/home/demo/public_html/domain1.com/log/*log /home/demo/public_html/domain2.com/log/*log {

If you have more log directories or files to add, just insert them into that list.

Rotate

Next we’ll want to make sure logrotate only keeps as many old logs as we want:

rotate 14

We’ll use 14 in this example to keep two weeks’ worth of logs, but you can of course adjust that number to something suitable to your requirements.

Interval

Now we’ll tell the web server to rotate these logs daily (again, change it if you prefer a longer interval):

daily

Size (optional)

If you prefer a weekly rotation it’s wise to specify a max log size as well, to be on the safe side. The max size setting doesn’t make much difference if you have the logs rotating daily, but if you use “weekly” or longer instead you might also include the line:

size 50M

That way if a log starts getting too large (from unexpectedly heavy traffic, for example) it will be rotated early rather than allowing it to get too unwieldy.

Compression

We’ll specify whether or not we want these logs to be compressed when they’re archived. For this example we’ll use delaycompress to account for the graceful restart of apache, which means we also need to turn compression on:

compress
delaycompress

Sharedscripts

You might have several virtual hosts, and that would mean several logs to rotate. To make sure the web server only gets restarted after all the rotations are done we add:

sharedscripts

Postrotate

We’ll specify a postrotate script that will restart the web server:

postrotate
        /usr/sbin/apache2ctl graceful > /dev/null
endscript

And finally, we close the config block with a curly bracket:

}

The whole shebang

Once we bring it all together our config file will look like this:

/home/demo/public_html/domain1.com/log/*log /home/demo/public_html/domain2.com/log/*log {
        rotate 14
        daily
        compress
        delaycompress
        sharedscripts
        postrotate
                /usr/sbin/apache2ctl graceful > /dev/null
        endscript
}

You’ll want to test that, of course, either by making sure you’re watching things when the nightly cron jobs are run, or by running logrotate right now:

/usr/sbin/logrotate /etc/logrotate.conf

If you don’t get any errors back you should be okay. But if you want to be absolutely certain you can run through some of the tests we would use when we suspect something isn’t working right.

Testing logrotate

If you suspect logrotate is having some trouble, or you just want to make sure a new config you’ve put in place will work, there are some useful flags you can pass to logrotate when you run it from the command line:

Verbose

The verbose flag, “-v”, tells logrotate to say what it’s doing while it’s doing it. It’s very useful when trying to find out why logrotate doesn’t rotate a log when you want it to.

Debug

The debug flag, “-d”, tells logrotate to go through the motions of rotating logs but not actually rotate them. It can be handy if you want to test a new config file but don’t want any actual log rotation run when you do (if you’re working on a production server, for example).

The debug flag is good for checking that the config file is formatted properly and that logrotate can find the log files it would rotate. However, since it doesn’t actually run the rotations it doesn’t test some parts of the process like the postrotate scripts.

Force

The force flag, “-f”, forces logrotate to rotate all logs when it runs, whether or not they would normally need to be rotated at that time. If you want to thoroughly test logrotate’s configs this is the flag to use. Just remember that logrotate will be rotating logs and deleting old ones according to the configuration you’ve set up, so don’t accidentally rotate out a recent log you needed to keep.

The force flag can be useful if you’re convinced that logrotate should be rotating a log, but it isn’t. Forcing the issue will help you tell if the problem is that logrotate doesn’t think the log needed rotating (if you run with the force flag and the log is rotated), or if the problem is that logrotate isn’t able to affect the log file (if you run it and nothing happens to the log).

Note that if logrotate is set to add a date to the name of an archived log, not even using the force flag will get logrotate to make a new archive in the same day (since the name it would use for the archive is already taken). In that circumstance you may need to rename the most recent archive (for each log file in a given config block) before you can force a log rotation.

Combining flags

The testing flags can be used together quite effectively. To have logrotate tell you what it would do if you made it rotate everything, but not actually rotate anything, you can combine all three:

/usr/sbin/logrotate -vdf /etc/logrotate.conf

You’ll get treated to a long list of things logrotate would do, including which log files it would rotate and what it would do during that process.

If you then want to test all the rotate configs in their entirety — including the scripts run after rotations — you can run logrotate without the debug flag:

/usr/sbin/logrotate -vf /etc/logrotate.conf

All the logs will be rotated, and skimming the output should help you catch any obvious problems. You’ll also want to make sure that all your services are still running okay (that there was nothing wrong with the postrotate scripts), and that all the logs actually did get rotated.

How logrotate remembers

If you find that a log isn’t rotating even though it’s old enough that it should, a simple approach to fixing the problem is to manually run logrotate with the “-f” flag. But if you’re the sort who wants to know why something’s gone wrong there’s one more file you can check before forcing a rotation:

/var/lib/logrotate/status

That file is where logrotate stores information about when it last rotated each log file. If you look inside that file you’ll see something like:

logrotate state -- version 2
"/var/log/acpid.log" 2010-6-18
"/var/log/iptables.log" 2010-6-18
"/var/log/uucp.log" 2010-6-29
...

It’s a straightforward format – the log file location is on the left, and the date when it was last rotated is on the right. Sometimes it can happen that the dates on your server get a little wonky (if you were tinkering with an NTP service or the like), and the date when a log was last rotated winds up being a future date. If that’s happened you’ll see it here.

If you want to check logrotate out with a particular log file but don’t want to force everything to rotate, you can delete the log’s entry from the logrotate status file. Then when you run logrotate normally, it should create a new entry for the log with today’s date (even though it may not actually rotate the log – it uses that first run as a baseline if it’s just interval-based).

Summary

For something that runs quietly in the background and only really performs one type of task, logrotate does quite a bit. You hopefully understand logrotate better than you wanted to. At the least you should be able to set up new logrotate config files for your own purposes, either creating them from scratch or copying existing configs and modifying them appropriately. And most importantly, you can keep your logs from getting out of control.

 

 

Logrotate is a great utility to manage and adminster log files in a system. It provides you with a lot of options, namely, automatic rotation, compression and mailing of log files. Each log file can be rotated hourly, daily, weekly or monthly. In this article, I will show you how to manage your log files using logrotate.

Any type of logs can be managed using logrotate. It runs as a daily cron job. These days most of the operating systems come with logrotate pre-installed so you probably won’t have to deal with the installation. Lets start with looking at the files and directories used by logrotate.

Files and Directories used by Logrotate

The 2 important files in case of logrotate are

/etc/logrotate.conf

/var/lib/logrotate.status    or    /var/lib/logrotate/status

The first file is where the whole configuration and default options of logrotate is stored. The second file contains the information about the last rotation times of different log files.

The directory thats usually used by logrotate is

/etc/logrotate.d

The options specified in /etc/logrotate.conf are used as default for rotating and managing log files. But if we want to specify separate settings for a set of log files then we can add their config files in this directory /etc/logrotate.d. Although we can even use another directory to save config files by changing an option in /etc/logrotate.conf.

Now, enough of the theory lets see how these config files really look and what all the different options in it mean.

Logrotate Options

Let me first show you some commonly used options of logrotate and then we will see how to configure various options in the config file.

rotate <num>	Log files are rotated “num” times before getting deleted or mailed.
daily	When used this means log files should be rotated daily
weekly	Log files should be rotated weekly
monthly	Log files should be rotated monthly
notifempty	Don’t rotate the log if its empty
compress	Compress the files after rotating them
delaycompress	This options means to delay the compression of a log file to the next rotation cycle. This is used in combination with compress.
missingok	If the log file is missing, go on to the next one without issuing an error message.
create <mode> <owner> <group>	After rotation create a new empty file with the following properties. e.g create 640 root adm. Just “create” will ensure to inherit the properties of previous log files.

Logrotate config files

As stated above, /etc/logrotate.conf is where the default options to handle logs are specified. A typical log file will look something like this.

# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp, or btmp — we’ll rotate them here
/var/log/wtmp {
    missingok
    monthly
    create 0664 root utmp
    rotate 1
}

/var/log/btmp {
    missingok
    monthly
    create 0660 root utmp
    rotate 1
}

The options mentioned in the config file above means that the logs are going to be rotated “weekly” and only 4 copies of backlogs will be maintained. On every rotation “create” a new empty file with properties inherited from the previous logs. Compression of backlogs is disabled.

The “include” directive is used to mention the directory in which the package specific log config files will be saved.

Managing Package specific log files in /etc/logrotate.d

Now, if there is an application which generates logs how would it be able to use logrotate to control and manage its logs. This is all done using its /etc/logrotate.d directory. This is where applications drop the information required by logrotate to manage their logs. e.g consider the example of apache logs. My apache logrotate config file looks something likes this (its /etc/logrotate.d/apache2).

/var/log/apache2/*.log {
    weekly
    missingok
    rotate 5
    compress
    delaycompress
    notifempty
    create 640 root adm
    sharedscripts
    postrotate
        if [ -f “`. /etc/apache2/envvars ; echo ${APACHE_PID_FILE:-/var/run/apache2.pid}`” ]; then
            /etc/init.d/apache2 reload > /dev/null
        fi
    endscript
}

Now, cross checking  the options from the table mentioned above the following properties of the logs can be inferred.

• Its going to be a weekly rotation
• 5 backlogs will be maintained in compressed format. But since we are using ‘delaycompress’ as well, the compression of a backlog will take place in the next rotation.
• No rotation will take place if file is empty.
• No error in case no log is found.
• On every rotation a new file will be created with owner root, group adm and permission 640

We haven’t discussed anything about ‘sharedscripts’ or ‘postrotate’ yet. Lets begin with prerotate/endscript and postrotate/endscript.

Prerotate and Postrotate

Both of these options control the scripts that will need to be run either before or after rotating a log file. Lets take the above example.

postrotate
        if [ -f “`. /etc/apache2/envvars ; echo ${APACHE_PID_FILE:-/var/run/apache2.pid}`” ]; then
            /etc/init.d/apache2 reload > /dev/null
        fi
endscript

‘endscript’ is mandatory to be mentioned in both prerotate and postrotate to specify the ending of the script.

You would have also noticed the sharedscripts option in the apache2 logrotate config file. Now, ifsharedscripts options is not specified, prerotate and postrotate will run the scripts on each log file which is rotated. But if you look carefully at the file logs mentioned in the apache2 logrotate conf file, its a wild card /var/log/apache2/*.log i.e. not just a single file but a bunch of files matching a pattern. These files in our case will be /var/log/apache2/access.log and error.log. So, for every rotation we will have 2 log files.

But look at the script that we want to run after rotation. It reloads the apache server and there is probably no need to run this script twice. So, by using sharedscripts we are ensuring that the script doesn’t run on every rotated log file but only once.