Linux mountpoint can not be unmounted no matter what?

Question

I have a mount point in Linux that does not show any open files in lsof or fuser. The mount point was exported in NFS, but is no longer exported, and there are no more files or directories in the mount point. We have checked to make sure no one is in the directory, going so far as to count and check the bash processes as well. At this point, we are absolutely stumped as to why we are unable to unmount the mount points.

[root@head-02 proc]# umount /PoolB/daily.2020.12.13
umount: /PoolB/daily.2020.12.13: target is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

We do not have the directory exported in NFS:

[root@head-02 proc]# showmount -e localhost
Export list for localhost:
/PoolB/LegacyExchange 172.18.10.0/24
/PoolB/Customers      172.18.10.0/24
/PoolB/Import         172.18.10.0/24

Output from FUSER:

[root@head-02 proc]# fuser -vm /PoolB/daily.2020.12.13
                 USER        PID ACCESS COMMAND
/PoolB/daily.2020.12.13:
                 root     kernel mount /PoolB/daily.2020.12.13

Output from LSOF:

[root@head-02 proc]# lsof +D /PoolB/daily.2020.12.13
[root@head-02 proc]#

The system is a ZFS On Linux system, so I could try to change the mount point there or delete it. The ultimate goal is to delete the filesystem from the ZPOOL:

[root@head-02 proc]# zfs set mountpoint=legacy PoolB/daily.2020.12.13
umount: /PoolB/daily.2020.12.13: target is busy.
        (In some cases useful info about processes that use
        the device is found by lsof(8) or fuser(1))
cannot unmount '/PoolB/daily.2020.12.13': umount failed
[root@head-02 proc]#

I even tried just deleting the filesystem:

[root@head-02 proc]# zfs destroy PoolB/daily.2020.12.13
umount: /PoolB/daily.2020.12.13: target is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
cannot unmount '/PoolB/daily.2020.12.13': umount failed
[root@head-02 proc]#

Out of desperation, here is the strace output of the umount command that fails:

[root@head-02 proc]# strace umount /PoolB/daily.2020.12.13
execve("/usr/bin/umount", ["umount", "/PoolB/daily.2020.12.13"], 0x7ffcd33fef18 /* 22 vars */) = 0
brk(NULL)                               = 0x55d4721b9000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4088778000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=22424, ...}) = 0
mmap(NULL, 22424, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f4088772000
close(3)                                = 0
open("/lib64/libmount.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\237\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=277808, ...}) = 0
mmap(NULL, 2370944, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f4088315000
mprotect(0x7f4088355000, 2097152, PROT_NONE) = 0
mmap(0x7f4088555000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x40000) = 0x7f4088555000
mmap(0x7f4088557000, 3456, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f4088557000
close(3)                                = 0
open("/lib64/libblkid.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\207\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=265600, ...}) = 0
mmap(NULL, 2358152, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f40880d5000
mprotect(0x7f4088111000, 2093056, PROT_NONE) = 0
mmap(0x7f4088310000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3b000) = 0x7f4088310000
mmap(0x7f4088314000, 2952, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f4088314000
close(3)                                = 0
open("/lib64/libuuid.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340\24\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=20064, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4088771000
mmap(NULL, 2113920, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f4087ed0000
mprotect(0x7f4087ed4000, 2093056, PROT_NONE) = 0
mmap(0x7f40880d3000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f40880d3000
close(3)                                = 0
open("/lib64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320i\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=155784, ...}) = 0
mmap(NULL, 2255184, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f4087ca9000
mprotect(0x7f4087ccd000, 2093056, PROT_NONE) = 0
mmap(0x7f4087ecc000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x23000) = 0x7f4087ecc000
mmap(0x7f4087ece000, 6480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f4087ece000
close(3)                                = 0
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20&\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2156160, ...}) = 0
mmap(NULL, 3985888, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f40878db000
mprotect(0x7f4087a9e000, 2097152, PROT_NONE) = 0
mmap(0x7f4087c9e000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c3000) = 0x7f4087c9e000
mmap(0x7f4087ca4000, 16864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f4087ca4000
close(3)                                = 0
open("/lib64/libpcre.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\25\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=402384, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4088770000
mmap(NULL, 2494984, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f4087679000
mprotect(0x7f40876d9000, 2097152, PROT_NONE) = 0
mmap(0x7f40878d9000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x60000) = 0x7f40878d9000
close(3)                                = 0
open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\r\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=19288, ...}) = 0
mmap(NULL, 2109712, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f4087475000
mprotect(0x7f4087477000, 2097152, PROT_NONE) = 0
mmap(0x7f4087677000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f4087677000
close(3)                                = 0
open("/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@m\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=142232, ...}) = 0
mmap(NULL, 2208904, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f4087259000
mprotect(0x7f4087270000, 2093056, PROT_NONE) = 0
mmap(0x7f408746f000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x7f408746f000
mmap(0x7f4087471000, 13448, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f4087471000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f408876f000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f408876e000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f408876c000
arch_prctl(ARCH_SET_FS, 0x7f408876c880) = 0
mprotect(0x7f4087c9e000, 16384, PROT_READ) = 0
mprotect(0x7f408746f000, 4096, PROT_READ) = 0
mprotect(0x7f4087677000, 4096, PROT_READ) = 0
mprotect(0x7f40878d9000, 4096, PROT_READ) = 0
mprotect(0x7f4087ecc000, 4096, PROT_READ) = 0
mprotect(0x7f40880d3000, 4096, PROT_READ) = 0
mprotect(0x7f4088310000, 12288, PROT_READ) = 0
mprotect(0x7f4088555000, 4096, PROT_READ) = 0
mprotect(0x55d470fcf000, 4096, PROT_READ) = 0
mprotect(0x7f4088779000, 4096, PROT_READ) = 0
munmap(0x7f4088772000, 22424)           = 0
set_tid_address(0x7f408876cb50)         = 37241
set_robust_list(0x7f408876cb60, 24)     = 0
rt_sigaction(SIGRTMIN, {sa_handler=0x7f408725f820, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7f40872685f0}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {sa_handler=0x7f408725f8b0, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f40872685f0}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
statfs("/sys/fs/selinux", 0x7fffc7932c50) = -1 ENOENT (No such file or directory)
statfs("/selinux", 0x7fffc7932c50)      = -1 ENOENT (No such file or directory)
brk(NULL)                               = 0x55d4721b9000
brk(0x55d4721da000)                     = 0x55d4721da000
open("/proc/filesystems", O_RDONLY)     = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4088777000
read(3, "nodev\tsysfs\nnodev\trootfs\nnodev\tr"..., 1024) = 350
stat("/etc/sysconfig/64bit_strstr_via_64bit_strstr_sse2_unaligned", 0x7fffc7932830) = -1 ENOENT (No such file or directory)
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0x7f4088777000, 4096)            = 0
access("/etc/selinux/config", F_OK)     = 0
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=106075056, ...}) = 0
mmap(NULL, 106075056, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f4080d2f000
close(3)                                = 0
getuid()                                = 0
geteuid()                               = 0
getuid()                                = 0
geteuid()                               = 0
getgid()                                = 0
getegid()                               = 0
prctl(PR_GET_DUMPABLE)                  = 1 (SUID_DUMP_USER)
lstat("/etc/mtab", {st_mode=S_IFLNK|0777, st_size=17, ...}) = 0
getuid()                                = 0
geteuid()                               = 0
getgid()                                = 0
getegid()                               = 0
prctl(PR_GET_DUMPABLE)                  = 1 (SUID_DUMP_USER)
stat("/run", {st_mode=S_IFDIR|0755, st_size=960, ...}) = 0
lstat("/run/mount/utab", 0x7fffc7932b50) = -1 ENOENT (No such file or directory)
mkdir("/run/mount", 0755)               = -1 EEXIST (File exists)
stat("/run/mount/utab", 0x7fffc7932a40) = -1 ENOENT (No such file or directory)
stat("/run/mount", {st_mode=S_IFDIR|0755, st_size=40, ...}) = 0
geteuid()                               = 0
getegid()                               = 0
getuid()                                = 0
getgid()                                = 0
access("/run/mount", R_OK|W_OK)         = 0
stat("/PoolB/daily.2020.12.13", {st_mode=S_IFDIR|0755, st_size=39, ...}) = 0
readlink("/PoolB", 0x7fffc79309b0, 4096) = -1 EINVAL (Invalid argument)
readlink("/PoolB/daily.2020.12.13", 0x7fffc79309b0, 4096) = -1 EINVAL (Invalid argument)
lstat("/etc/mtab", {st_mode=S_IFLNK|0777, st_size=17, ...}) = 0
open("/proc/self/mountinfo", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4088777000
read(3, "18 41 0:18 / /sys rw,nosuid,node"..., 1024) = 1024
stat("/etc/sysconfig/64bit_strstr_via_64bit_strstr_sse2_unaligned", 0x7fffc7930370) = -1 ENOENT (No such file or directory)
read(3, "rfs efivarfs rw\n29 25 0:25 / /sy"..., 1024) = 1024
read(3, "ev,noexec,relatime shared:19 - c"..., 1024) = 1024
read(3, ",xattr,posixacl\n131 128 0:42 / /"..., 1024) = 1024
read(3, " - zfs PoolB/daily.2020.12.13 rw"..., 1024) = 145
readlink("/proc", 0x7fffc7930830, 4096) = -1 EINVAL (Invalid argument)
readlink("/proc/self", "37241", 4096)   = 5
readlink("/proc/37241", 0x7fffc7930830, 4096) = -1 EINVAL (Invalid argument)
readlink("/proc/37241/mountinfo", 0x7fffc7930830, 4096) = -1 EINVAL (Invalid argument)
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0x7f4088777000, 4096)            = 0
getuid()                                = 0
geteuid()                               = 0
getgid()                                = 0
getegid()                               = 0
prctl(PR_GET_DUMPABLE)                  = 1 (SUID_DUMP_USER)
stat("/run", {st_mode=S_IFDIR|0755, st_size=960, ...}) = 0
stat("/run/mount/utab", 0x7fffc79328c0) = -1 ENOENT (No such file or directory)
stat("/sbin/umount.zfs", 0x7fffc7931a80) = -1 ENOENT (No such file or directory)
stat("/sbin/fs.d/umount.zfs", 0x7fffc7931a80) = -1 ENOENT (No such file or directory)
stat("/sbin/fs/umount.zfs", 0x7fffc7931a80) = -1 ENOENT (No such file or directory)
stat("/run/mount/utab", 0x7fffc7932b30) = -1 ENOENT (No such file or directory)
umount2("/PoolB/daily.2020.12.13", 0)   = -1 EBUSY (Device or resource busy)
open("/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2502, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4088777000
read(3, "# Locale name alias data base.\n#"..., 4096) = 2502
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0x7f4088777000, 4096)            = 0
open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/util-linux.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US.utf8/LC_MESSAGES/util-linux.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US/LC_MESSAGES/util-linux.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.UTF-8/LC_MESSAGES/util-linux.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.utf8/LC_MESSAGES/util-linux.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en/LC_MESSAGES/util-linux.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, "umount: ", 8umount: )                 = 8
write(2, "/PoolB/daily.2020.12.13: target "..., 153/PoolB/daily.2020.12.13: target is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))) = 153
write(2, "\n", 1
)                       = 1
close(1)                                = 0
close(2)                                = 0
exit_group(32)                          = ?
+++ exited with 32 +++
[root@head-02 proc]#

As you can see, the strace reveals the system call umount2 fails with EBUSY, which in the man page for that system call, just says "The Target is Busy".

We even deleted all the contents of that filesystem with rm -rf /PoolB/daily.2020.12.13/ which ran successfully. We don't have swap spaces on it, and it isn't listed in /etc/exports.

We are at a loss at this point, and for obvious reasons, rebooting the system is for all intents and purposes a non-starter. We should not have to reboot anyway just because a filesystem has to be unmounted. Any advice would be greatly appreciated.

Linux Kernel version is 3.10.0-1062.4.3.el7.x86_64 ZFS version is 0.8.2-1.el7 as is the zfs-dkms package.

Thank you for reading this far.

This link might have some useful advice (including a ZFS scenario that on the surface sounds very similar to yours): https://unix.stackexchange.com/questions/15024/umount-device-is-busy-why — Brandon Xavier, Mar 04 '21 at 10:33
I have done (almost) all of those suggestions to make sure im not doing something stupid. The only thing I can not do is restart my NFS server because this system servers a large multi-tenant cloud's VMs. I have made sure the mount points are not exported, and the NFSD PID is not showing up in the fuser for this mount point. LSOF shows nothing as well. — user1955162, Mar 04 '21 at 16:56

Sleepwalk3r · Answer 1 · 2021-03-04T11:48:35.633

0

Maybe an obvious question but you didn't mention it in your post.
Did you try and use the force option with umount (umount -f)?
It is specifically used for umounting stale/unreachable network filesystems like NFS (so probably not exactly your case).
You can also try the lazy umount if force doesn't help.

from the man page:

-f, --force
    Force an unmount (in case of an unreachable NFS system)
-l, --lazy
    Lazy unmount.  Detach the filesystem from the file
    hierarchy now, and clean up all references to this
    filesystem as soon as it is not busy anymore.

edited Mar 04 '21 at 11:48

answered Mar 04 '21 at 11:37

Sleepwalk3r

9
1

My goal is to remove the filesystem (because its size is growing and growing as the system holds its underlying snapshot data larger and larger). When I lazy unmount it, it wont let me delete it. Force does nothing for me, the straces are identical. Force yo my understanding just works for unreachable NFS shares. – user1955162 Mar 04 '21 at 16:54

Linux mountpoint can not be unmounted no matter what?

1 Answers1