At times, I come across the following message in syslog:
\nkernel: EXT3-fs warning (device dm-3): ext3_dx_add_entry: Directory index full!<\/p>\n
On systems in which this is being constantly logged to syslog, this can become rather annoying, but it is not a bug. It’s informing the user that somewhere within the system, a directory has more than 32,000 subdirectories. A directory can have at most 31998 subdirectories, because an inode can have at most 32000 links. (See external link below for source)<\/p>\n
Sometimes, this can be difficult to find, especially in a case whereby a physical server is hosting a couple of hundres VPS accounts. To help break down where the culprit(s) is coming from, I’ve put together this quick script to tell me where to start looking:<\/p>\n
#!\/bin\/bash
\n# Kristian Reese
\n# http:\/\/kb.kristianreese.com<\/p>\n
for veid in $(vzlist -aHh 0*.tld.com -o veid)
\ndo
\n echo $veid
\n vzctl exec $veid ‘find \/var -type d 2>\/dev\/null |<\/p>\n
(
\n while read dir; do cnt=`ls -l “$dir” | wc -l`<\/p>\n
if [ “$cnt” -gt “10000” ]; then
\n echo “$cnt: $dir”
\n fi<\/p>\n
done
\n )’<\/p>\n
echo
\ndone
\nMost of the time, customer emails are queued up in their qmail mailboxes. I’ll investigate to see what these emails are. For example, in one particular case, I found the customer was running a cronjob to check the status of httpd every waking moment of uptime. The crontab did not output to \/dev\/null, and therefore emails were being sent each and everytime the crontjob ran. This script was named “keep-apache-alive” and was subjected as such in the emails, so I looked to see how many of these there were in the users \/var\/qmail\/mailnames\/domain.com\/info\/Maildir\/new. There were a LOT, so I blew them away and fixed cron to output to \/dev\/null, preventing the cronjob from sending emails.<\/p>\n
-bash-3.1# find . -type f -exec cat {} \\; | grep “keep-apache-alive” | wc -l
\n-bash-3.1# find . -type f -exec grep -q “keep-apache-alive” {} \\; -exec rm {} \\; -exec echo removed {} \\;
\n-bash-3.1# find . -type f -exec cat {} \\; | grep Subject | more<\/p>\n
Another method would be to use the following bit of perl code:<\/p>\n
#!\/usr\/bin\/perl
\nuse File::Find;
\nmy %dirs = ();
\nfind
\n(
\n sub
\n {
\n unless(-d $_) { $dirs{$File::Find::dir}++; }
\n }, ‘.’
\n);<\/p>\n
foreach(keys %dirs) {
\n if ($dirs{$_} > 20000) {
\n print “$_ has $dirs{$_} files.\\n”;
\n }
\n}<\/p>\n","protected":false},"excerpt":{"rendered":"
At times, I come across the following message in syslog: kernel: EXT3-fs warning (device dm-3): ext3_dx_add_entry: Directory index full!<\/p>\n
On systems in which this is being constantly logged to syslog, this can become rather annoying, but it is not a bug. It’s informing the user that somewhere within the system, a directory has more than […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,17],"tags":[],"_links":{"self":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/4168"}],"collection":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4168"}],"version-history":[{"count":1,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/4168\/revisions"}],"predecessor-version":[{"id":4169,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/4168\/revisions\/4169"}],"wp:attachment":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4168"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4168"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4168"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}