17

Background

I had a small logrotate misshap... Logrotate would rotate the archived logs by misstake causing a quadratic growth of files in my /var/log/. And by the time I caught wind that something was awry, /var/log/ already contained a few million files...

I managed to (after some hairloss and find/sed/grep magic) remove all offending files and fix my logrotate config. And thought all was well...

Problem

Whenever I ls/du -hs or otherwise list the contents of /var/log/ (which now contains 80mb of archives/logs and at most a few hundred files) the process doing that hangs for a good minute or two. I do believe this is somehow related to the logrotate misshap but I'm not certain, it could be something else. Anyway I'm at loss to where to start debugging or looking for a fix for this. Please help :3

Other info

uname -a
Linux xxx 3.3.8-gentoo #18 SMP Sat Sep 21 22:44:40 CEST 2013 x86_64 Intel(R) 
Core(TM)2 CPU 4400 @ 2.00GHz GenuineIntel GNU/Linux

cat /proc/meminfo 
MemTotal:        2051552 kB
MemFree:           75612 kB
Buffers:            9016 kB
Cached:          1740608 kB
SwapCached:            0 kB

CFQ IO scheduler + SLUB allocator

I thought this: How many files in a directory is too many? (Downloading data from net) was related but I don't have the files left anymore.

Edit

The problem persists even after a call to init 1 so I think it is safe to assume there is no other process to blame but the FS.

Solution (as applied from accepted answer)

init 1
mv /var/log /var/log1
mkdir /var/log
chmod --reference=/var/log1 /var/log
chown --reference=/var/log1 /var/log
tar -C /var/log1 -cvp . | tar -C /var/log -xvp
rm -rf /var/log1
init 5
Community
  • 1
Emma
  • 374
  • 3
  • 11

2 Answers2

26

Directories only ever grow in size, not shrink. Try moving all those files out into a a temporary directory (like log2) then rmdir the old directory and rename the temp one as the new permanent one.

Matthew Ife
  • 22,927
  • 2
  • 54
  • 71
  • I just did this as you answered. It worked :) Sadly, I don't have enough reputation to upvote you :/ – Emma Sep 22 '13 at 17:42
1

From man e2fsck:

-D     Optimize directories in filesystem.  This option causes e2fsck to try to optimize all  directories,
either  by  reindexing  them if the filesystem supports directory indexing,  or by sorting and com‐
pressing directories for smaller directories, or for filesystems using traditional linear  directo‐
ries.

Even without the -D option, e2fsck may sometimes optimize a few directories --- for example, if di‐
rectory indexing is enabled and a directory is not indexed and would benefit from being indexed, or
if the index structures are corrupted and need to be rebuilt.  The -D option forces all directories
in the filesystem to be optimized.  This can sometimes make them  a  little  smaller  and  slightly
faster to search, but in practice, you should rarely need to use this option.

The  -D  option  will  detect  directory  entries with duplicate names in a single directory, which
e2fsck normally does not enforce for performance reasons.

In other words, if you have a file system volume you can afford to take offline you can simply run e2fsck -Df <block_device> on it and it will shrink all the directories. Including the root directory in the file system, which you can't delete otherwise without re-formatting the volume. In my case it successfully shrunk the root directory on the volume from something like 56MiB (used to have more than 1M of files) to about 220KiB (about 3-4K of files).

moonwalker
  • 111
  • 1