1

I am using a CentOS 7 machine as a BackupPC server to pull in backups from my two frontend servers once per night. Last night the backups failed so I checked the server to see where the problem is. During my SSH session I was confronted with error messages regarding my quota/resource limit was exceeded. I couldn't restart any service, I couldn't create a file or folder etc.

[admin@host ~]$ df
Dateisystem    1K-Blöcke Benutzt Verfügbar Verw% Eingehängt auf
/dev/vzfs      314572800 9919068 304653732    4% /
devtmpfs         2097152       0   2097152    0% /dev
tmpfs            2097152       0   2097152    0% /dev/shm
tmpfs            2097152     128   2097024    1% /run
tmpfs            2097152       0   2097152    0% /sys/fs/cgroup

Well, plenty of space left on my drive. Let's check the inodes

[admin@host ~]$ df -i
    Dateisystem                  Inodes             IBenutzt      IFrei IUse% Eingehängt auf
    /dev/vzfs      18446744070451413169 18446744069415392088 1036021081  100% /
    devtmpfs                     524288                   58     524230    1% /dev
    tmpfs                        524288                    1     524287    1% /dev/shm
    tmpfs                        524288                  152     524136    1% /run
    tmpfs                        524288                   10     524278    1% /sys/fs/cgroup

That's a crazy number but I am using reiserfs which is not using inodes/adjusts inodes dynamically. (correct me if I am wrong)

[admin@host ~]$ df -T
Dateisystem    Typ      1K-Blöcke Benutzt Verfügbar Verw% Eingehängt auf
/dev/vzfs      reiserfs 314572800 9919068 304653732    4% /
devtmpfs       devtmpfs   2097152       0   2097152    0% /dev
tmpfs          tmpfs      2097152       0   2097152    0% /dev/shm
tmpfs          tmpfs      2097152     128   2097024    1% /run
tmpfs          tmpfs      2097152       0   2097152    0% /sys/fs/cgroup

To get the system somehow running again I deleted 4 backups and re-run df -i

[admin@host ~]$ df -i
Dateisystem    Inodes IBenutzt  IFrei IUse% Eingehängt auf
/dev/vzfs      800000   792553   7447  100% /
devtmpfs       524288       58 524230    1% /dev
tmpfs          524288        1 524287    1% /dev/shm
tmpfs          524288      150 524138    1% /run
tmpfs          524288       10 524278    1% /sys/fs/cgroup

Pretty cool, everything is working again and I can create files/folders and start services again. So, I started another backup while observing the inodes count and once the magic number of 800000 is reached it immediately explodes to this crazy count above and the quota exceeded error msgs are back. However my backup doesn't include any folders or files that large.

Now my question(s): Is this raise of inodes number expected behavior of reiserfs? And why is my system running out of quota/resource space? Is there any relation?

Here is my quota output:

[admin@host ~]$ quota
Dateisystemquotas für user admin (uid 1000): kein
[admin@host ~]$ quota -g
Dateisystemquotas für group wheel (gid 10): kein
Dateisystemquotas für group admin (gid 1000): kein
travisbotello
  • 23
  • 1
  • 7
  • 1
    This looks like a problem with the `/dev/vzfs` device or a limitation/quota on that rather on the `reiserfs` or some sort of interplay. – Thomas Feb 24 '17 at 11:21
  • 2
    You really shouldn't use ReiserFS anymore, and certainly not on a new(ish) deployment with CentOS 7. Better options exist. – Sven Feb 24 '17 at 11:23
  • @Sven Even when using this CentOS 7 system with mainly with BackupPC? I thought reiserfs would be the best option here... -.- – travisbotello Feb 24 '17 at 11:32
  • 1
    No, ReiserFS is essentially dead and isn't developed anymore. XFS is a good alternative, and ZFS is particularly nice for backups due to its snapshot functionality (though I don't know if/how you can make this useful with BackupPC). The same should be true for `btrfs` but I've never used this and am not fully aware about its current status on CentOS. – Sven Feb 24 '17 at 11:37
  • @Thomas Any ideas on how to debug/identify those limitations/problems? – travisbotello Feb 24 '17 at 11:37
  • Reinstall, and this time leave the filesystem at its default (XFS). – Michael Hampton Feb 24 '17 at 11:41
  • Well, I assume you are running a container with vzfs. So you should go to your host and investigate with `vz...` commands like `vzquota` and/or `vzctl`. If you do not host the container, you definitely should contact your hoster. – Thomas Feb 24 '17 at 13:01
  • @Thomas That's true...I am running a container with vzfs. That's why I can't switch the filesystem. I am forced to use reiserfs which I assumed would work fine with BackupPC. I just contacted the hoster who is now investigating. – travisbotello Feb 24 '17 at 13:23

0 Answers0