Risks of removing fuse from Linux (to get rid of .fuse_hidden files)

Question

What are the side effects of removing fuse from Ubuntu server 16.04?

Each time I want to remove a movie from this machine, it just gets renamed to .fuse_hidden<some_long_number>.

Stopping the movie server (Plex) does not solve the problem, the only way to actually get rid of the file is to reboot. I have tried using lsof to locate the service using the file to no avail.

If I remove fuse (sudo apt-get remove --auto-remove fuse) what are some of the pitfalls? Will my system become unstable?

By removing fuse, will deleting the file have the expected results? i.e. will the file go away?

Answer 1

Short answer

Removing fuse will probably cause inability to mount the filesystem.

FUSE is Filesystem in Userspace which means there is a userspace program that handles all operations performed on this specific filesystem (while non-fuse filesystem support works in kernelspace). It may be ntfs-3g in your case; even if it's not, the story is generally the same.

All specific FUSE implementations depend on the fuse package and ntfs-3g is among them (well, formally it's pre-dependency but it makes no difference here). This means you cannot remove fuse and have ntfs-3g (or another FUSE program) working.

What really bothers you is the existence of .fuse_hidden files (compare: XY problem). The rest of my answer addresses this issue.

---

The context

It looks like you can ignore .fuse_hidden files, as stated here: What is a .fuse_hidden file and why do they exist?

The answer to How to delete .fuse_hidden files? compares FUSE behavior to NFS:

This is similar to what happens when you delete a file that another system has open on a NFS mount.

and NFS behavior is explained here. From therein:

What happens when a file is opened by a client, and deleted? The file needs to keep having name, so that the client that has it open can still access it. But when a file is deleted, it is expected that no more file by that name exists afterwards. So NFS servers turn the deletion of an open file into a renaming: the file is renamed to .nfs… (.nfs followed by a string of letters and digits).

It's all because a file in Linux can be deleted while it's open by a process. This mechanism works by design with inode-based local filesystems (like ext family) but it needs to be emulated somehow if access to files depends on their names only. I believe the situation with NTFS is somewhat complicated. You can find some interesting comments and links under the link above.

Well, ntfs-3g could mimic Windows general behavior and deny removing the file when it's in use. The problem is many Linux programs expect they can remove the file they still use. It's pretty clever.

Let's suppose your program needs a temporary file. It creates one, opens it and removes immediately – and Linux does allow this. From now on the task of actually freeing the disk space (when the file is not needed anymore) is somebody else's job: the kernel or FUSE. This task will be handled gracefully even if your program dies unexpectedly or gets forcefully killed.

On the other hand, if your program cannot remove the file beforehand, it's still its job to clean when it's done; unexpected termination may leave the "abandoned" temporary file. And what if somebody else opened the same file? Then your program cannot remove it even when it's done and everything is fine with it.

It's good to keep this Linux way of handling files. Files like .fuse_hidden or .nfs are the cost of this philosophy and they will be removed eventually. But let's say something went wrong and they won't. It's still relatively easy to spot them during manual maintenance, while in Windows you may have "abandoned" files and not know it. Linux way seems to me as much more tidier approach.

---

Some tests

My testbed:

# whoami
root
# cat /etc/issue
Ubuntu 16.04.2 LTS \n \l
# uname -a
Linux foobar 4.4.0-59-generic #80-Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
# dpkg -l | grep ntfs-3g
ii  ntfs-3g                                     1:2015.3.14AR.1-1ubuntu0.1 amd64                      read/write NTFS driver for FUSE

Preparing mountpoints:

# mkdir /mnt/ext4 /mnt/ntfs

Preparing filesystems:

# truncate -s 20M image-ext4
# truncate -s 20M image-ntfs
# mkfs.ext4 -Fq image-ext4 
# mkfs.ntfs -FqQ image-ntfs

(Chatty output of mkfs.ntfs omited.)

Mounting:

# mount image-ext4 /mnt/ext4/
# mount image-ntfs /mnt/ntfs/

Initial disk usage:

# df -h /mnt/ext4/ /mnt/ntfs/
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0       19M  172K   17M   1% /mnt/ext4
/dev/loop1       20M  2.5M   18M  13% /mnt/ntfs

Creating files:

# dd if=/dev/urandom bs=1M count=10 | tee /mnt/ext4/file > /mnt/ntfs/file
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.645865 s, 16.2 MB/s

Disk usage:

# df -h /mnt/ext4/ /mnt/ntfs/
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0       19M   11M  6.8M  60% /mnt/ext4
/dev/loop1       20M   13M  7.6M  63% /mnt/ntfs

Opening files, then removing:

# exec 3<> /mnt/ext4/file 
# exec 4<> /mnt/ntfs/file 
# rm /mnt/ext4/file /mnt/ntfs/file

Disk usage:

# df -h /mnt/ext4/ /mnt/ntfs/
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0       19M   11M  6.8M  60% /mnt/ext4
/dev/loop1       20M   13M  7.6M  63% /mnt/ntfs

So despite the removal the diskspace is still in use. This is because the files are still open.

Actual files:

# ls -A /mnt/ext4/ /mnt/ntfs/
/mnt/ext4/:
lost+found

/mnt/ntfs/:
.fuse_hidden0000000200000001

At this point I make copies of the filesystems (for later comparison). I know in general I shouldn't do this while they are mounted but the idea is to simulate hard reset before I close the files. Still I want the copied filesystems clean, hence the sync command. Additionally the --reflink=always option allows me to make snapshot-like copies on my Btrfs filesystem where image-ext4 and image-ntfs are stored; in this test plain cp should work as well.

# sync
# cp --reflink=always image-ext4 copy-ext4
# cp --reflink=always image-ntfs copy-ntfs

I can check if copy-ext4 is clean:

# fsck.ext4 copy-ext4
e2fsck 1.42.13 (17-May-2015)
copy-ext4: clean, 11/5136 files, 1849/20480 blocks

Unfortunately there is no fsck.ntfs.

Let's continue with original filesystems. Closing the files:

# exec 3>&-
# exec 4>&-

Disk usage:

# df -h /mnt/ext4/ /mnt/ntfs/
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0       19M  172K   17M   1% /mnt/ext4
/dev/loop1       20M  2.5M   18M  13% /mnt/ntfs

Contents:

# ls -A /mnt/ext4/ /mnt/ntfs/
/mnt/ext4/:
lost+found

/mnt/ntfs/:

The .fuse_hidden file is no more and the diskspace is free again. The file disappears when it's no longer needed.

Let's see what happens after the simulated reset, when the files were not closed properly. Mounting copies:

# umount /mnt/ext4 /mnt/ntfs 
# mount copy-ext4 /mnt/ext4/
# mount copy-ntfs /mnt/ntfs/

Disk usage:

# df -h /mnt/ext4/ /mnt/ntfs/
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0       19M  172K   17M   1% /mnt/ext4
/dev/loop1       20M   13M  7.6M  63% /mnt/ntfs

Actual files:

# ls -A /mnt/ext4/ /mnt/ntfs/
/mnt/ext4/:
lost+found

/mnt/ntfs/:
.fuse_hidden0000000200000001

So this is a scenario when you should manually remove .fuse_hidden file(s). Note that if ntfs-3g didn't create such a file and denied removal in the first place, you would now have a leftover file with its old name; you would have it even without reset and this means even more maintenance.

I believe inode-based filesystems don't need such a maintenance at all.

Cleaning:

# umount /mnt/ext4 /mnt/ntfs
# rmdir /mnt/ext4/ /mnt/ntfs/
# rm image-ext4 copy-ext4 image-ntfs copy-ntfs