At times, I come across the following message in syslog:
kernel: EXT3-fs warning (device dm-3): ext3_dx_add_entry: Directory index full!
On systems in which this is being constantly logged to syslog, this can become rather annoying, but it is not a bug. It’s informing the user that somewhere within the system, a directory has more than 32,000 subdirectories. A directory can have at most 31998 subdirectories, because an inode can have at most 32000 links. (See external link below for source)
Sometimes, this can be difficult to find, especially in a case whereby a physical server is hosting a couple of hundres VPS accounts. To help break down where the culprit(s) is coming from, I’ve put together this quick script to tell me where to start looking:
#!/bin/bash
# Kristian Reese
# http://kb.kristianreese.com
for veid in $(vzlist -aHh 0*.tld.com -o veid)
do
echo $veid
vzctl exec $veid ‘find /var -type d 2>/dev/null |
(
while read dir; do cnt=`ls -l “$dir” | wc -l`
if [ “$cnt” -gt “10000” ]; then
echo “$cnt: $dir”
fi
done
)’
echo
done
Most of the time, customer emails are queued up in their qmail mailboxes. I’ll investigate to see what these emails are. For example, in one particular case, I found the customer was running a cronjob to check the status of httpd every waking moment of uptime. The crontab did not output to /dev/null, and therefore emails were being sent each and everytime the crontjob ran. This script was named “keep-apache-alive” and was subjected as such in the emails, so I looked to see how many of these there were in the users /var/qmail/mailnames/domain.com/info/Maildir/new. There were a LOT, so I blew them away and fixed cron to output to /dev/null, preventing the cronjob from sending emails.
-bash-3.1# find . -type f -exec cat {} \; | grep “keep-apache-alive” | wc -l
-bash-3.1# find . -type f -exec grep -q “keep-apache-alive” {} \; -exec rm {} \; -exec echo removed {} \;
-bash-3.1# find . -type f -exec cat {} \; | grep Subject | more
Another method would be to use the following bit of perl code:
#!/usr/bin/perl
use File::Find;
my %dirs = ();
find
(
sub
{
unless(-d $_) { $dirs{$File::Find::dir}++; }
}, ‘.’
);
foreach(keys %dirs) {
if ($dirs{$_} > 20000) {
print “$_ has $dirs{$_} files.\n”;
}
}
Recent Comments