0

Almost at a loss here. Been running a fast server for a couple of years with Plesk. Hosting our own website and services. Over the last few days, I noticed a lot of problems but didn't think to check available file space. Checking this last night, I get:

[root@server ~]# df -H
Filesystem      Size  Used Avail Use% Mounted on
rootfs           22G   21G     0 100% /
/dev/root        22G   21G     0 100% /
devtmpfs         34G  238k   34G   1% /dev
/dev/sda2       136G  7.4G  121G   6% /var
tmpfs            34G     0   34G   0% /dev/shm
/dev/root        22G   21G     0 100% /var/named/chroot/etc/named
/dev/sda2       136G  7.4G  121G   6% /var/named/chroot/var/named
/dev/root        22G   21G     0 100% /var/named/chroot/etc/named.rfc1912.zones
/dev/root        22G   21G     0 100% /var/named/chroot/etc/rndc.key
/dev/root        22G   21G     0 100% /var/named/chroot/usr/lib64/bind
/dev/root        22G   21G     0 100% /var/named/chroot/etc/named.iscdlv.key
/dev/root        22G   21G     0 100% /var/named/chroot/etc/named.root.key

I am fairly knowledgeable about server systems, but cannot get any answers as to what is filling this space up, how to remove it. Or, increase the rootfs size; if that's even possible. Related to this, maybe, is that my backups are GONE! on a mounted NAS. Been using rsync and a custom msqlbackup script to do this, but operations on the server have been a bit haywire over the last few days. Any pointers to what I can do? I have searched the other posts here and on other websites, but none that can help me identify what I can do. I really would appreciate your assistance. Thanks

Some further information: Output of fdisk -l:

Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        2612    20971520   83  Linux
/dev/sda2            2612       19324   134244352   83  Linux
/dev/sda3           19324       19390      525312   82  Linux swap / Solaris

Contents of /etc/fstab:

/dev/sda1       /       ext4    errors=remount-ro       0       1
/dev/sda2       /var    ext4    defaults        0       2
/dev/sda3       none    swap    defaults        0       0
proc            /proc   proc    defaults                0       0
sysfs           /sys    sysfs   defaults                0       0
tmpfs           /dev/shm        tmpfs   defaults        0       0
devpts          /dev/pts        devpts  defaults        0       0
10.16.101.3:/nas-000009/mininas-001783 /mnt nfs rw 0    0

After going into recovery and removing some log files to clear a wee bit of space, the output of df -h is oddly different:

Filesystem      Size  Used Avail Use% Mounted on
rootfs           20G   20G     0 100% /
/dev/root        20G   20G     0 100% /
devtmpfs         32G  232K   32G   1% /dev
/dev/sda2       126G  6.6G  113G   6% /var
tmpfs            32G     0   32G   0% /dev/shm
/dev/root        20G   20G     0 100% /var/named/chroot/etc/named
/dev/sda2       126G  6.6G  113G   6% /var/named/chroot/var/named
/dev/root        20G   20G     0 100% /var/named/chroot/etc/named.rfc1912.zones
/dev/root        20G   20G     0 100% /var/named/chroot/etc/rndc.key
/dev/root        20G   20G     0 100% /var/named/chroot/usr/lib64/bind
/dev/root        20G   20G     0 100% /var/named/chroot/etc/named.iscdlv.key
/dev/root        20G   20G     0 100% /var/named/chroot/etc/named.root.key

rootfs is now only 20GB in size, 100% usage. Dynamically changing on contents?

Mark Carson
  • 41
  • 4
  • 1
    What do you mean by haywire? Are there any relevant messages in your logs ? – user9517 Nov 08 '15 at 16:46
  • The `used` column is zero for `tmpfs` on the `df -H` output you posted. Your root filesystem is full, though. When's the last time you looked in `/var/log`? – Andrew Henle Nov 08 '15 at 16:50
  • Haywire? On Thursday, I logged into Plesk to find that the license key needed reloading. Done this to discover that multiple cronjobs were firing - maybe 12 rsyncs for backups. All cores 100%. It maybe related. Messages in my logs were limited to firewall logging... – Mark Carson Nov 08 '15 at 17:07
  • There was a particularly large log file in /var/log, but only around 300MB. Tend to keep a close eye on these – Mark Carson Nov 08 '15 at 17:08
  • /var/log and other domain logs total about 500MB space. – Mark Carson Nov 08 '15 at 17:22
  • /var is only partially used and it is not an issue here as it is a separate filesystem. – user9517 Nov 08 '15 at 18:02
  • Odd that my server has only 27GB of storage space actually used. – Mark Carson Nov 09 '15 at 07:34
  • I won't pretend to know all there is about Unices, but it seems very strange to me that you'd have the same filesystem mounted many times to different locations, or that you'd have multiple filesystems mounted to the same location (unless you're using an overlay FS for the latter, but I don't think that's even how they're used). – GregL Nov 09 '15 at 12:41
  • @GregL It only looks strange until you realize that it is a few files and directories bind mounted into a chroot. – kasperd Nov 10 '15 at 12:08
  • 1
    @kasperd, consider me schooled! – GregL Nov 10 '15 at 12:46

1 Answers1

1

Some help from Iain identified that the mounted networked attached storage had gone away. All the backups were being written to the /rootfs filling it very quickly. I'll just need to figure out why that happened and ensure that it doesn't happen again.

Mark Carson
  • 41
  • 4