Best way to prevent the root system filling up when a mount fails?

Question

We have an internal web server (virtualized, hosting ReviewBoard, but not super relevant) and we have a relatively consistent failure mode with failed NFS mounts causing / to fill up. Distro is Ubuntu (don't ask) if a solution depends on a different distribution, it will be slower to implement.

Backups are being performed to /mnt/backup/, which is supposed to an NFS mount to another system. Unfortunately, when the mount fails, or drops off, backups get performed on the root filesystem, which as you can imagine doesn't take long before / is full, and then services start to fail.

A number possible solutions have been discussed.

Monitor /mnt/backups and ensure it's not root. Perhaps a cron job.
Use /mnt/protected/backups, and mount /protected first to a small filesystem, perhaps a loop mount to a local file so it is much less likely to fail.
Chmod a-rwx /mnt/backups (the root filesystem mount point). I'm not sure if mounting over protected director will work, I think it does.
On the mounted tree create a directory called "Backups", then soft link "ln - s /mnt/backup/Backups /Backups". Using /Backups for backups will fail unless the /mnt/backup is mounted, since the local tree doesn't contain the sub-directory.
Performing a check that the directory is correctly mounted in the backup script.

I'm interested in any feedback on these approaches, pros cons or any additional techniques people use as a standard way of protecting the root file system from this type of nastiness.

score 25 · Answer 1 · answered Dec 05 '11 at 16:15

25

The most error-proof solution is to make the mount point unwritable. This would be your solution #3. However there is one additional step you should perform. chattr +i /mnt/backups. This is because even with no permissions, root would still be able to write to the directory. With chattr +i (sets immutable flag) not even root can write to it. Once the mount is mounted, the permissions dont matter as the permissions will be of the remote directory, not the local one.

answered Dec 05 '11 at 16:15

phemmer

5,789
2
26
35

1

That's a pretty neat trick - never thought of using 'chattr' – warren Dec 05 '11 at 18:08
1

I use this technique on all my mount points. – 3dinfluence Dec 05 '11 at 20:38
2

I tried this with `encfs` a fuse filesystem. It gives error: `fusermount: user has no write access to mountpoint` – ctrl-alt-delor Jun 22 '16 at 08:10
This is the solution I normally use, and I think it should be the accepted answer. – shodanshok Aug 16 '17 at 20:08

ewwhite · Accepted Answer · 2012-03-13T09:41:16.637

15

Number 5 - Put a test in your backup script to ensure that the directory is mounted before continuing. The script should fail if the mount is not available or present. Or you can just make sure things are mounted prior to running the backup.

Try the mountpoint command, which checks if a specified directory is a mountpoint:

mountpoint -q /mnt/backups || mount /mnt/backups

edited Mar 13 '12 at 09:41

answered Dec 05 '11 at 02:31

ewwhite

194,921
91
434
799

Hmm, I guess I'd add || echo "Mount /mnt/backups failed" 2>&1 Or perhaps just exist there. Anyway, thanks!!! – Peter Dec 05 '11 at 04:47

score 3 · Answer 3 · answered Dec 05 '11 at 02:40

What ewwhite said. Also, some extra monitoring for basis system health wouldn't be a bad idea.

Something like Monit can check to see how much space is left. If you want to go full bore on system monitoring, you can look at Nagios, but Monit is light weight and will do the basics.

Since you're using Ubuntu, Monit is already in repo, so you can do "sudo apt-get install monit", then start looking at the configuration files to tell it to send alerts to the right place, monitor the right services, etc. Here's a quick tutorial.

score 1 · Answer 4 · edited Aug 16 '17 at 23:29

1

Here is a one liner that you can run as a cron job, it assumes the mount in question is in fstab:

if mountpoint -q /mnt ; then : ; else mount /mnt ; fi

edited Aug 16 '17 at 23:29

peterh

4,914
13
29
44

answered Aug 16 '17 at 18:32

dan

11
1

score 0 · Answer 5 · answered Mar 13 '12 at 11:33

For a longer term solution: Not sure how to do this on Ubuntu (I am RH centric) or if it is worth while (if you have just one machine) but a methodology that has worked for us for MANY years is to create separate logical volumes, file systems even volume groups on server machines. So as a matter of standard practice we create LVM logical volumes for /, /tmp. /usr, /usr/local, /opt, /home, /var, swap space and a separate partition for /boot. This approach makes it extra difficult for file systems to fill up and disable the system. Actually this approach will make it almost impossible to fill up the / file system. You still need to watch /tmp, /var of course. If we need to house data then we create a totally different volume group for it. This approach has other benefits too, expand file systems at will, move them around, create new ones etc. And as a historical note we carried over this method to Linux from the AIX OS back in the 1990's!

So your solution to preventing the filesystem filling up if a mount fails is to add more mounts? What? — phemmer, Mar 13 '12 at 12:03

Best way to prevent the root system filling up when a mount fails?

5 Answers5

Linked