6

As part of an automated VM creation system a block device is mounted to a temporary folder ( /tmp/whatever ) . Various scripts install and configure the VM prior to it's first run.

Recently something changed, the temporary mount is busy and refuses to umount. In trying to determine what might still be keeping a file open I've checked:

Tests run as root

  • mount
  • lsof | grep /tmp/
  • fuser -m /tmp/...
  • exportfs -rv
  • Restarting the daemon which runs the creation scripts anyway...
  • ps axf
  • dmsetup table
  • losetup -a
  • fuser -vm /tmp/tmp.random-chars/ (yields two lines)
    • USER PID ACCESS COMMAND
    • /tmp/tmp.random-chars: root kernel mount /tmp/tmp.random-chars

None of the above tests have results that point to filesystem use, however umount -f still complains "Device or resource busy" / "device is busy".

What other tests should I try so that I can get to the true root cause and thus hopefully fix the stuck mount without reboot on a system I can't presently restart for a while as well as prevent this from reoccurring?

It is also /doubtful/ (but I don't know how to check) that kernel modules from the temporary mount are loaded, as the temporary mount has a different version of Linux installed than the host is running.

edits

  • From various search results it appears that /modules/ are simply read in to memory. I am unaware of if the kernel could have open files and of how to access any such list.
  • Added the dmsetup / losetup to the lists of "tests that don't show issues"
  • fuser -vm as suggested in freenode ##linux
Michael J. Evans
  • 173
  • 1
  • 1
  • 5

3 Answers3

6

If it's part of a build process, I'm assuming you'll need to reboot at some point anyway. Try inserting a "lazy" unmount into the process. Use umount -l /tmp and see if that helps you get past this barrier in the process.

ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • umount -fl was tried the last time this occurred, it does not solve the issue, it only hides it from view. – Michael J. Evans Apr 25 '14 at 18:58
  • 1
    can you show what you are doing in your build vm process? – c4f4t0r Apr 25 '14 at 19:04
  • Several customizations to the default ganeti deployment process, including installing new debian packages within the chroot. I did a thing that (at least previously) prevents the daemon start/stop events from occurring while in this state. However this should be approached from a more general 'something is holding open the resource, locate the thing' approach and that approach would also be more useful for anyone else who runs across this page in the future. – Michael J. Evans Apr 25 '14 at 19:21
0

We got the very same problem but it appears it only happens if the root filesystem of the virtual machine is ext4. ext3 works correctly. I know it sounds strange but it might be a kernel bug described in http://www.phoronix.com/scan.php?page=news_item&px=MTIxNDQ

If this is so, we must simply wait for kernel patch OR avoid installing new vm in main machine. Installing it from a temporary linux run as virtual machine works correctly because the machine is rebooted without rebooting the main machine.

Did you try ext3? If not, try installing at ext3, you can convert it to ext4 later if using ext4 is crucial (and it's described in http://www.debian-administration.org/article/643/Migrating_a_live_system_from_ext3_to_ext4_filesystem)

ChewbaccaKL
  • 228
  • 3
  • 10
0

One reason umount can fail is if the underlying IP address of the remote device has changed.

I have seen this happen when I remote-mount a laptop from my desktop server. The first mount takes place with IP address A. Although rebooting the laptop gives it address B, my desktop continues to record address A as the address of the laptop. I can see this when I examine the IP address returned by the mount command and comparing it with the current address of the laptop.

  • I was able to unmount using umount -l
  • The solution to this problem was -- for me -- to use a fixed IP address for the laptop
Moshe Yudkowsky
  • 21
  • 3