loop device does not auto detach on umount

Question

I have a backup script in which I umount a squashfs file, I do backup then mount it back again. A simplified version of the script is:

umount /home/backup/auto/mnt/os
mksquashfs /src-dir /home/backup/auto/os.sqfs.img
mount -t squashfs -o loop /home/backup/auto/os.sqfs.img /home/backup/auto/mnt/os

The idea is to have access at all times to backedup data.

And this worked for long time. However I got an error of not having enough loopback devices and I saw that umount does not detach loop devices mounted with mount anymore.

Lately I had two major changes:

1. I updated the kernel to 2.6.32-754.11.1.el6.i686
2. I moved the OS to an SSD.

I just noticed that there is new kernel update 2.6.32-754.12.1.el6, so I updated, but it did not fix the problem.

OS: CentOS 6.10 i686

Update: This is on the faulty system:

# strace -e trace=ioctl,mount mount -t squashfs -o loop os.sqfs.img mnt/os
ioctl(3, LOOP_GET_STATUS, {number=0, offset=0, flags=0, name="/home/squash/web.img", ...}) = 0
ioctl(3, LOOP_GET_STATUS, {number=0, offset=0, flags=0, name="/home/squash/web.img", ...}) = -1 ENXIO (No such device or address)
ioctl(4, LOOP_SET_FD, 0x3)              = 0
ioctl(4, LOOP_SET_STATUS64, {offset=0, number=0, flags=0, file_name="/home/backup/auto/os.sqfs.img", ...}) = 0
mount("/dev/loop1", "mnt/os", "squashfs", MS_MGC_VAL, NULL) = 0
+++ exited with 0 +++

# cat /etc/mtab|grep 'mnt/os'
/dev/loop1 /home/backup/auto/mnt/os squashfs ro,relatime 0 0

# ls -la /etc/mtab
lrwxrwxrwx. 1 root root 12 Apr 17 02:53 /etc/mtab -> /proc/mounts

# notice that here there is no even the umount call. Am I missing something?
# strace -e trace=ioctl,umount,mount umount mnt/os
+++ exited with 0 +++

And this is on another system with the same version of everything, but it's 64 bit:

# strace -e trace=ioctl,mount mount -t squashfs -o loop os.sqfs.img mnt/os
ioctl(3, LOOP_GET_STATUS, {number=0, offset=0, flags=0, name="/storage/backup/auto/home.sqfs.img", ...}) = 0
ioctl(3, LOOP_GET_STATUS, {number=1, offset=0, flags=0, name="/storage/backup/auto/mail.sqfs.img", ...}) = 0
ioctl(3, LOOP_GET_STATUS, {number=1, offset=0, flags=0, name="/storage/backup/auto/mail.sqfs.img", ...}) = -1 ENXIO (No such device or address)
ioctl(4, LOOP_SET_FD, 0x3)              = 0
ioctl(4, LOOP_SET_STATUS64, {offset=0, number=0, flags=0, file_name="/storage/backup/auto/os.sqfs.img", ...}) = 0
mount("/dev/loop2", "mnt/os", "squashfs", MS_MGC_VAL, NULL) = 0
+++ exited with 0 +++

# strace -e trace=ioctl,umount,mount umount mnt/os
umount("/storage/backup/auto/mnt/os", 0) = 0
ioctl(3, LOOP_CLR_FD)                   = 0
+++ exited with 0 +++

# losetup -a
/dev/loop0: [0904]:35656625 (/storage/backup/auto/home.sqfs.img)
/dev/loop1: [0904]:35656626 (/storage/backup/auto/mail.sqfs.img)

I don't see LO_FLAGS_AUTOCLEAR on the good system too. but I do see difference in umount.

I reinstalled util-linux-ng - it didn't help. There are no updates for this package and it has never been updated. Maybe it's related to the kernel. I wander why it's working just fine on the 64 bit OS. I might install a 32 bit VM to test if it'll do the same.

-- FIX --

The problem was that I made /etc/mtab a link to /proc/mounts during system copy using a live CD of CentOS 7 and I forgot about it. I had to re-install grub, so I chroot to the os and made the link because grub-install wasn't working. I forgot to restore it. I fixed this by:

rm -f /etc/mtab && cat /proc/mounts |grep -v rootfs > /etc/mtab

So it works now:

root@home auto# smount os.sqfs.img mnt/os
root@home auto# losetup -a
/dev/loop0: [0816]:7090072 (/home/squash/web.img)
/dev/loop1: [0816]:7864675 (/home/backup/auto/os.sqfs.img)
/dev/loop4: [0816]:7864569 (/home/backup/auto/web.sqfs.img)
root@home auto# umount mnt/os
root@home auto# losetup -a
/dev/loop0: [0816]:7090072 (/home/squash/web.img)
/dev/loop4: [0816]:7864569 (/home/backup/auto/web.sqfs.img)

Answer 1

This is what I've gathered by poking around the source for util-linux-ng 2.17.2, which was included with CentOS 6:

On kernels < 2.6.37, Linux requires that an entry be written to /etc/mtab for loop mounted filesystems in order to autoclear the loop device later. If /etc/mtab is not writable at mount time or the entry was removed from /etc/mtab then the loop device won't be autocleared. On kernels >= 2.6.37 the kernel remembers the loop device backing store path and it is not required to be in /etc/mtab.

I was able to reproduce this on CentOS 6.10 by making /etc/mtab immutable, with chattr +i /etc/mtab. Simply deleting it wasn't enough; it would be recreated with at least a partial mount table. This is not something that should happen in normal operation, so if you find this file has been set immutable you should look for an administrator mistake or a security compromise.

If /etc/mtab doesn't exist or is incomplete on your older Linux system, you can make a copy from /proc/mounts. Don't do this on modern Linux systems though, where /etc/mtab is instead a symbolic link to /proc/self/mounts left in place for backward compatibility with ancient code that expects a mount table at that location.

Answer 2

TL;DR: it appears not to be a kernel issue, but the mount command not behaving as expected by not setting the loop device's LO_FLAGS_AUTOCLEAR flag. Using a more recent out-of-distribution mount command should probably solve it.

---

Stracing a (recent system's) command mount, led me to loop(4)'s flag LO_FLAGS_AUTOCLEAR:

LO_FLAGS_AUTOCLEAR (since Linux 2.6.25)
    The loopback device will autodestruct on last close.

So my "modern" mount command issues successfully this ioctl as seen there:

# strace -e trace=ioctl,mount mount -o loop /tmp/block.img /mnt/
ioctl(3, LOOP_CTL_GET_FREE)             = 0
ioctl(4, LOOP_SET_FD, 3)                = 0
ioctl(4, LOOP_SET_STATUS64, {lo_offset=0, lo_number=0, lo_flags=LO_FLAGS_AUTOCLEAR, lo_file_name="/tmp/block.img", ...}) = 0
ioctl(3, BLKGETSIZE64, [1073741824])    = 0
ioctl(3, CDROM_GET_CAPABILITY, 0)       = -1 EINVAL (Invalid argument)
ioctl(3, BLKSSZGET, [512])              = 0
mount("/dev/loop0", "/mnt", "ext4", MS_MGC_VAL, NULL) = 0
+++ exited with 0 +++
# umount /mnt
# losetup -l
#

(The losetup syntax here differs from CentOS6).

I thought if the issue is not in the kernel (since OP's 2.6.32 > 2.6.25), it could be from the mount command. To check, I installed a centos6-i686 LXC container, installed strace, created the missing /dev/loop* files and ran a totally privilegied shell inside (using lxc-attach's -e option) allowed to perform mount operations:

# lxc-attach -e -n centos6-i686
[root@centos6-i686 ~]# cat /etc/centos-release 
CentOS release 6.10 (Final)
# dd if=/dev/zero of=/tmp/block.img bs=1 count=1 seek=$((2**30-1))
[...]
# ls -lh /tmp/block.img 
-rw-r--r--. 1 root root 1.0G Apr 22 19:06 /tmp/block.img
# mkfs.ext4 /tmp/block.img 
[...]
# strace -e trace=ioctl,mount mount -o loop /tmp/block.img /mnt/
ioctl(3, LOOP_GET_STATUS, {number=0, offset=0, flags=0, name="", ...}) = -1 ENXIO (No such device or address)
ioctl(4, LOOP_SET_FD, 0x3)              = 0
ioctl(4, LOOP_SET_STATUS64, {offset=0, number=0, flags=0, file_name="/tmp/block.img", ...}) = 0
ioctl(3, BLKGETSIZE64, 1073741824)      = 0
ioctl(3, CDROM_GET_CAPABILITY or SNDRV_SEQ_IOCTL_UNSUBSCRIBE_PORT, 0) = -1 EINVAL (Invalid argument)
mount("/dev/loop0", "/mnt/", "ext4", MS_MGC_VAL, NULL) = 0
+++ exited with 0 +++
# umount /mnt
# losetup --show /dev/loop0
/dev/loop0: [003f]:1756539 (/tmp/block.img)
# losetup -d /dev/loop0
#

Clearly the ioctl wasn't issued with the clear flag and this caused the loop device leak. Looking at the sources, it appears this feature has been around for a long time, and is present in util-linux 2.17.2 (CentOS6's version). CentOS6's man mount even tells:

Since Linux 2.6.25 is supported auto-destruction of loop devices and then any loop device allocated by mount will be freed by umount independently on /etc/mtab.

So I don't know why it didn't work, and it appears to be a bug (or this could be related to my environment: 64bits kernel 5.0.x etc.).