At times, I come across the following message in syslog:
kernel: EXT3-fs warning (device dm-3): ext3_dx_add_entry: Directory index full!

On systems in which this is being constantly logged to syslog, this can become rather annoying, but it is not a bug. It’s informing the user that somewhere within the system, a directory has more than 32,000 subdirectories. A directory can have at most 31998 subdirectories, because an inode can have at most 32000 links. (See external link below for source)

Sometimes, this can be difficult to find, especially in a case whereby a physical server is hosting a couple of hundres VPS accounts. To help break down where the culprit(s) is coming from, I’ve put together this quick script to tell me where to start looking:

#!/bin/bash
# Kristian Reese
# http://kb.kristianreese.com

for veid in $(vzlist -aHh 0*.tld.com -o veid)
do
echo $veid
vzctl exec $veid ‘find /var -type d 2>/dev/null |

(
while read dir; do cnt=`ls -l “$dir” | wc -l`

if [ “$cnt” -gt “10000” ]; then
echo “$cnt: $dir”
fi

done
)’

echo
done
Most of the time, customer emails are queued up in their qmail mailboxes. I’ll investigate to see what these emails are. For example, in one particular case, I found the customer was running a cronjob to check the status of httpd every waking moment of uptime. The crontab did not output to /dev/null, and therefore emails were being sent each and everytime the crontjob ran. This script was named “keep-apache-alive” and was subjected as such in the emails, so I looked to see how many of these there were in the users /var/qmail/mailnames/domain.com/info/Maildir/new. There were a LOT, so I blew them away and fixed cron to output to /dev/null, preventing the cronjob from sending emails.

-bash-3.1# find . -type f -exec cat {} \; | grep “keep-apache-alive” | wc -l
-bash-3.1# find . -type f -exec grep -q “keep-apache-alive” {} \; -exec rm {} \; -exec echo removed {} \;
-bash-3.1# find . -type f -exec cat {} \; | grep Subject | more

Another method would be to use the following bit of perl code:

#!/usr/bin/perl
use File::Find;
my %dirs = ();
find
(
sub
{
unless(-d $_) { $dirs{$File::Find::dir}++; }
}, ‘.’
);

foreach(keys %dirs) {
if ($dirs{$_} > 20000) {
print “$_ has $dirs{$_} files.\n”;
}
}

Sometimes, after performing VPS migrations, the soft links in /etc/vz/conf may not be removed. Over time, the dead links may build up and cause many ‘No such file or directory’ messages when trying to look for common settings amonst all conf files (ie cat /etc/vz/conf/*.conf | grep whatever).
Quick easy way to delete these dead links:

~# cd /etc/vz/conf
~# for f in `find . -type l ! -exec test -r {} \; -print`; do rm -f $f; done

One liner:

~# find -L -type l -exec rm -f {} \;
* Please test before issuing these commands on production systems!!

Check if /etc/shadow exists. If it doesn’t, run the command ‘pwconv’ then try changing the user password again.

How to disable cron based mail alerts
The easiest way to disable cron based email alerts is to append the following to the specified command within the users crontab:

>/dev/null 2>&1

Example:

# m h dom mon dow command
59 23 * * * /root/scripts/hotfixes/vzagentdb.sh >/dev/null 2>&1
CRONDARGS=’-m off’ in /etc/sysconfig/crond may also be specified to turn off mail completely. Other postings have also indicated setting the MAILTO=”” option in the users crontab also works.

Note that I’ve seen cases where the qmail queue was so high that even after implementing these measures, it seemed to not work, but that was because the qmail queue was draining. Once it completely drained, the madness stopped.

System hangs during messagebus startup
This problem has been experienced after rebooting a system in which ldap has been configured. Note that LDAP functions properly immeidately after it is setup on the machine, but hangs at “staring system message bus” after a reboot. To fix, reboot in single user mode or into linux rescue and edit /etc/ldap.conf

If linux rescue is used, the disk may have to be remounted rw in order to edit /etc/ldap.conf if for some reason a user receives a read-only file system message or cannot chroot to /mnt/sysimage.

edit entry with following content:

nss_initgroups_ignoreusers root,ldap,named,avahi,haldaemon,dbus