Why does rm over NFS not free space immediately?

Question

I'm on a SUSE Enterprise server which has mounted a drive via an NFS mount. I have created a 34GB file via dd if=/dev/Zero of=test_file bs=8k Count=4194304 to test the write speed of the NFS share. After that I deleted the file via rm test_file and then I immediately used df.

The value in the used column of the output of df starts to decrease. I update by calling df every ~10 seconds and every time I see that the used value has decreased by ~2GB.

So it seems that the rm test_file command doesn't seem to have completed it's job although I have the command prompt available again (thus rm must have returned "0", right?). What's going on here in the background?

Answer 1

The filesystem on the remote node that's providing the backend storage for the NFS share isn't freeing the space immediately.

Strictly speaking, POSIX semantics don't require removal of a file to immediately free the space it was using (they actually don't require it to free any space at all, which is kind of important for some use cases). Because of this, and the fact that it may take a long time to free the space, some filesystems don't wait for the deallocation to happen, and just return once they can ensure that the file entry will not show up in directory listings (which is all that POSIX requires).

Possible reasons that it may take a while to free space include:

• The filesystem may scrub empty space as it's freed to prevent information disclosure. This is almost always an expensive operation.
• The filesystem may be on an SSD or thinly provisioned storage and configured to inform lower layers that the space is now not in use. This operation can also take quite a long time.
• The filesystem may need to update data structures for internal accounting purposes. This may or may not take a long time, but is usually not trivial.
• The filesystem may need to update internal data structures to handle any reflinks or other shared regions in the file properly.