How can a partition be full if du does not show it is?

1

On one of my systems, the root partition is full:

snip:˜ # df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs           11G  9.3G     0 100% /
devtmpfs        744M   36K  744M   1% /dev
tmpfs           751M     0  751M   0% /dev/shm
tmpfs           751M  296K  751M   1% /run
/dev/sda7        11G  9.3G     0 100% /
tmpfs           751M     0  751M   0% /sys/fs/cgroup
tmpfs           751M  296K  751M   1% /var/lock
tmpfs           751M  296K  751M   1% /var/run
tmpfs           751M     0  751M   0% /media
/dev/sda5       151M   39M  104M  28% /boot
/dev/sda8       4.4G  207M  3.3G   6% /home

But du does not show near 9.3 gigabyte of usage:

snip:~ # du /* -s -h
5.2M    /bin
34M /boot
36K /dev
22M /etc
199M    /home
154M    /lib
20M /lib64
0   /media
0   /mnt
0   /opt
0   /proc
7.9M    /root
288K    /run
7.1M    /sbin
0   /selinux
756K    /srv
0   /sys
0   /tmp
1.6G    /usr
1.1G    /var

It only accounts for about 3 gigabytes.

  • How can that be?
  • Where should I look for the remaining 6+ gigabytes of used gigabytes?

I'm using openSUSE 12.2:

snip:~ # cat /etc/SuSE-release
openSUSE 12.2 (x86_64)
VERSION = 12.2
CODENAME = Mantis

Jeroen Wiert Pluimers

Posted 2013-09-08T16:11:40.467

Reputation: 2 373

1Did you unlink any files? (e.g. deleting/clearing a log file while it is still in use. Thus removing it from the directory listing while it still occupies disk space?) – Hennes – 2013-09-08T16:15:09.133

Other idea: Did you use space in a directory on /, and then mounted another directory on top of that, thus masking the old files? – Hennes – 2013-09-08T16:16:08.217

@Hennes no, and no. I'm baffled: I've never seen anything like this happen on a linux system before. – Jeroen Wiert Pluimers – 2013-09-08T16:18:31.607

Lets nuance the previous comment @Hennes: I've not done consciously. Can there be any subsystem that might have done this? – Jeroen Wiert Pluimers – 2013-09-08T16:24:48.040

It is easily tested by rebooting (to free unlinked files) and then booting into single user mode. The second can also be done in other ways, e.g. tellinit 1, maybe shutdown without -h -r (not sure about that one on linux, on BSD that drops you to runlevel 1), umount -a. Or just looking at what is mounted and testing that. – Hennes – 2013-09-08T16:27:17.057

Your mounts might simply be "hiding" a lot of stuff underneath them. – Mat – 2013-09-08T16:33:39.030

I'm going to boot into single user mode and see what is going on. – Jeroen Wiert Pluimers – 2013-09-08T16:46:44.697

1@Hennes I booted a rescue system from CD as it would not come up properly in single user mode because root was full. From the rescue system, I did mount /dev/sda7 /mnt, then chroot /mnt, then du -s -h * again: virtually the same result: the du results don't add up to the df result. – Jeroen Wiert Pluimers – 2013-09-08T17:42:29.233

2Any luck with an fsck (before mounting RW) ?. Also, feel free to add that to the post so we can delete the comments. – Hennes – 2013-09-08T17:44:49.820

I found it through the openSUSE forums: it uses btrfs and snapshots. So the snapshots take up a lot of space. And I need to find out a way to delete old snapshots. https://forums.opensuse.org/english/get-technical-help-here/install-boot-login/490199-how-can-partition-full-if-du-does-not-show.html#post2583491

– Jeroen Wiert Pluimers – 2013-09-08T17:59:47.807

I think I found it: http://www.nrtm.org/index.php/2012/03/13/the-joys-of-btrfs-and-opensuse-or-no-space-left-on-device/

– Jeroen Wiert Pluimers – 2013-09-08T18:09:48.160

@Hennes found it and posted an answer. – Jeroen Wiert Pluimers – 2013-09-08T19:08:00.153

Answers

1

First some background information

If you have btrfs on your root file system on http://www.opensuse.org/en/, then two things will happen:

  1. openSUSE will automagically start using snapper to take snapshots of your root file system.
  2. the snapshots will take up diskspace that du doesn't show

This means that you will run out of disk space sooner than you'd expect. So the recommendation (not in the docs) is to make partitions that use snapshots twice as large as you normally would.

I have not found a way to show the size per snapshot or the total size of all snapshots.

So you have to monitor your free disk space with df or this btrfs specific command for the root (/) file system:

btrfs filesystem df /

Cleaning up snapper snapshots

Thanks to NerdyRoom™ » The joys of btrfs and OpenSuSE – or “no space left on device”. I found out the easiest way to delete older snapshots that you might want to delete (and you have to when you run out of disk space).

First run snapper list to see the sequence number of snapshots that are there.

From that list, select a reasonable lower and upper bound of snapshots to delete.

Then run this with the lower (1) and upper (3656) bound:

for i in `seq 1 3656`; do snapper delete $i; done

Edit 20161212:

An anonymous user suggested an edit to make this shorter. I agree, as the above can be done shorter as per snapper man page:

snapper delete 1-3656

Jeroen Wiert Pluimers

Posted 2013-09-08T16:11:40.467

Reputation: 2 373