Convert Ext3 to Ext4 Filesystem failed because I cannot umount drive

Question

I want to convert my Filesystem from Ext3 to Ext4 because of 32000 directory limit of Ext3. But when I want to umount the drive I get this error:

umount: /: device is busy
umount: /: device is busy

I tried following command to understand what make the drive busy:

root@cl-t159-101cl [~]# ps auxw|grep 538
root      8426  0.0  0.0   4888   700 pts/2    S+   10:04   0:00 grep 538

It's actually my remote session (I connect to my server using SSH) which do not allow to umount the drive.

I also tried command fuser to see processes which are using the drive:

root@cl-t159-101cl [~]# fuser -mu /dev/sda5
/dev/sda5:               1rce(root)     2rc(root)     3rc(root)     4rc(root)     5rc(root)     6rc(root)     7rc(root)     8rc(root)     9rc(root)    10rc(root)    11rc(root)    12rc(root)    13rc(root)    14rc(root)    15rc(root)    16rc(root)    17rc(root)    18rc(root)    19rc(root)    25rc(root)    26rc(root)    27rc(root)    28rc(root)    29rc(root)   166rc(root)   167rc(root)   168rc(root)   169rc(root)   172rc(root)   174rc(root)   252rc(root)   253rc(root)   254rc(root)   255rc(root)   256rc(root)   257rc(root)   258rc(root)   259rc(root)   420rc(root)   466rc(root)   467rc(root)   468rc(root)   469rc(root)   470rc(root)   476rc(root)   477rc(root)   478rc(root)   479rc(root)   480rc(root)   481rc(root)   482rc(root)   507rc(root)   540rce(root)  1136rc(root)  1504rc(root)  1523rc(root)  1524rc(root)  1525rc(root)  1526rc(root)  1527rc(root)  1548rc(root)  1550rc(root)  1790rc(root)  1870rc(root)  1886rc(root)  1887rc(root)  1889rc(root)  1894rc(root)  1900rc(root)  1902rc(root)  1903rc(root)  1904rc(root)  1910rc(root)  1932rce(root)  1940rce(root)  1941rce(root)  2357rce(root)  2360rce(root)  2374rce(root)  2422rce(root)  2437rce(root)  2461rce(mysql)  2574rce(mailnull)  2605rce(root)  2607rce(root)  2608rce(root)  2623rce(dovecot)  2624rce(dovecot)  2625rce(dovecot)  2626rce(dovecot)  2632rce(root)  2633rce(root)  2666rce(root)  2668rce(root)  2695rce(root)  2697rce(root)  2710rce(root)  2714rce(root)  2941rce(root)  2980rce(root)  2992rce(root)  2999rce(root)  3001rce(root)  3357rce(root)  3360rce(root)  3361rce(root)  3362rce(root)  3363rce(root)  3366rce(root)  3368rce(root)  3511rce(named)  3686rce(root)  3714rce(r)  3715rce(r)  5080rce(nobody)  5771rce(nobody)  5929rce(nobody)  5936rce(nobody)  6060rce(root)  6136rce(r)  6147rce(r)  6530rce(nobody)  6540rce(nobody)  6944rce(nobody)  6984rce(nobody)  6995rce(nobody)  7254rce(nobody)  7534rce(nobody)  7535rce(nobody)  7542rce(nobody)  7700rce(nobody)  7739rce(nobody)  7748rce(nobody)  7751rce(nobody)  7773rce(nobody)  7793rce(nobody)  7798rce(nobody)  7799rce(nobody)  7801rce(nobody)  7900rce(nobody)  7929rce(nobody)  8172rce(nobody)  8174rce(nobody)  8178rce(nobody)  8201rce(nobody)  8308rce(root)  8355rce(root)  8446rce(nobody)  8447rce(nobody)  8451rce(nobody)  8455rce(nobody)  8459rce(nobody)  8471rce(nobody)  8473rce(nobody)  8474rce(nobody)  8478rce(nobody)  8480rce(nobody)  8481rce(nobody)  8535rce(nobody)  8541rce(nobody)  8555rce(nobody)  8557rce(nobody)  8559rce(nobody)  8563rce(nobody)  8582rce(nobody)  8583rce(nobody)  8584rce(nobody)  8585rce(nobody)  8589rce(nobody)  8591rce(nobody)  8675rce(nobody)  8690rce(nobody)  8691rce(nobody)  8721rce(nobody)  8752rce(nobody)  8754rce(nobody)  8756rce(nobody)  8777rce(nobody)  8784rce(nobody)  8785rce(nobody)  8786rce(nobody)  8787rce(nobody)  8790rce(nobody)  8796rce(nobody)  9076rce(nobody)  9085rce(nobody)  9101rce(nobody)  9103rce(nobody)  9154rce(nobody)  9160rce(nobody)  9370rce(nobody)  9395rce(nobody)  9414rce(nobody)  9416rce(nobody)  9418rce(nobody)  9420rce(nobody)  9439rce(nobody)  9440rce(nobody)  9441rce(nobody)  9444rce(nobody)  9446rce(nobody)  9448rce(nobody)  9449rce(nobody)  9450rce(nobody)  9451rce(nobody)  9452rce(nobody)  9454rce(nobody)  9604rce(nobody)  9605rce(nobody)  9606rce(nobody)  9607rce(nobody)  9624rce(nobody)  9625rce(nobody)  9628rce(nobody)  9705rce(nobody)  9714rce(nobody)  9715rce(nobody)  9719rce(nobody)  9721rce(nobody)  9722rce(nobody)  9728rce(nobody)  9730rce(nobody)  9790rce(nobody)  9876rce(nobody)  9877rce(nobody)  9878rce(nobody)  9882rce(nobody)  9883rce(nobody)  9884rce(nobody)  9886rce(nobody)  9906rce(nobody)  9907rce(nobody)  9909rce(nobody)  9910rce(nobody)  9912rce(nobody)  9915rce(nobody)  9927rce(nobody)  9928rce(nobody)  9929rce(nobody)  9930rce(nobody)  9931rce(nobody)  9933rce(nobody)  9934rce(nobody)  9936rce(nobody)  9939rce(nobody)  9940rce(nobody)  9941rce(nobody)  9942rce(nobody) 10014rce(nobody) 10016rce(nobody) 10018rce(nobody) 10019rce(nobody) 10020rce(nobody) 10021rce(nobody) 10026rce(nobody) 10028rce(nobody) 10030rce(nobody) 10031rce(nobody) 10034rce(nobody) 10039rce(nobody) 10051rce(nobody) 10052rce(nobody) 10054rce(nobody) 10056rce(nobody) 10058rce(nobody) 10060rce(nobody) 10062rce(nobody) 10064rce(nobody) 10066rce(nobody) 10067rce(nobody) 10070rce(nobody) 10071rce(nobody) 10072rce(nobody) 10073rce(nobody) 10074rce(nobody) 10308rce(nobody) 10311rce(nobody) 10821rce(nobody) 11011rce(nobody)

Is there a solution for this problem? Is it possible to convert the filesystem at system startup when no process started?

Answer 1

Simplest solution would be to run frmo a bootable iso and change it from there, but "32000 directory limit" - FFS!

Answer 2

Changing the filessystem used by your root volume can be a hairy process (so make sure you have up-to-date tested backups before trying at all).

The usual recommendation is to do this sort of thing from a seperate boot, i.e. boot from a live CD so you are not using the OS that you are about to change the root filesystem of. As you say it is a remote machine this might not be possible.

You could probably make a custom initrd image that performed the changes for you in the next boot, but this is not a process for a beginner and can easily leave you with an unusuable machine if you don't have physical access to sort things out if they go wrong (and if you have physical access then it is safer+easier to go the live-CD route). During the boot process while the initrd is active your normal root is not mounted so you can modify it in ways that this would normally block, but you also can not remotely interact with the machine to deal with any errors in the process unless you have a KVMoIP setup (this usually costs extra, so if you don't know if you have one available then you don't have one available).

What sort of machine is this? If it is a VM hostred by somewhere like linode you can do the side-by-side boot (like using a live CD) by creating a new boot image and giving it access to the old one to make modifications. Depending on how much is on the machine you could update your backups, ask the host to rebuild with the alternate filesystem option, and resore the backups to this fresh install. This is a lot safer than trying to make an initrd that will do the conversion. If you don't have a good backup arrangement that you can use for this, now is the time to make one.

Why is the 32,000 objects-per-directory limit a problem for you? Is it your own code or some utility that is creating that many objects? Addressing this may be easier than messing around with the current filesystem. The answers to this question may help you there.