"Disk quota exceeded" when writing to /tmp, but plenty of space (linux)

Question

i have a VPS. It's managed with the notorious parallels plesk.

today i started seeing messages (via wordpress at first, but also from the command line shell), saying: "Disk quota exceeded".

user@machine:~$ echo aaa > /tmp/aaa
-bash: /tmp/aaa: Disk quota exceeded

but there's lots of space on the machine, and only 1 partition.

user@machine:~$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/vzfs             100G   24G   77G  24% /

I deleted all files from /tmp/* but still.

I don't know anything about "vzfs", maybe maybe it's the culprit?

What could've gone wrong? and how can to fix it?

Solution!

(if you DON'T use Parallels Plesk, see @Adalee's answer)

See @deltik 's solution, relevant to Parallel's system, as in my case.

df -i immediately gave a rediculous number of inodes: 18,446,744,069,620,218,961 which is insane for a wordpress website.

i further explored and found qmail's queue is full with inodes (it full with some hacker's attemtps to use my machine to spam, and failure replies)

fixing qmail isn't relevant here, but my workaround is relevant:

1. stopped qmail's service
2. i wasn't able to download qmhandle to the over-quota system, so i had to
3. extract and upload the script to another website,
4. delete the queue with this command:
5. perl <(wget -O - http://link.to.my/script.pl) -D

this way the script is running directly from the web (which is not a safe practice, but i had put the script there by myself), with command line options.

thanks @deltik for knowing and promptly recognizing this peculiar setup on the spot, and providing means to further diagnose !!!

Answer 1

You have a VZFS filesystem, which means that your VPS is a Parallels Virtuozzo virtual machine. In Virtuozzo the hosting provider can set limits on many parameters, including what allocations you get with VZFS.

Cause: Out of Inodes (Most Common)

After years of working with hundreds of Virtuozzo VPS customers who have had the issue of being unable to create files, even though there appeared to be plenty of free space, the vast majority of them had reached their inode limit. Run this command to see the inode allocation (Inodes), inodes used (IUsed), remaining inodes (IFree), and the percentage of inodes used (IUse%):

df -i

Having 100% inode usage happens a lot. Common causes in my experience:

• Spam email bouncebacks
• Outbound spam emails queued
• A lot of inbound emails stored
• Some user setting their PHP session garbage collection (session.gc_maxlifetime) to over a hundred years
• Way too many general cache files
• Object cache enabled in the WordPress plugin W3 Total Cache
• Magento error log (a new file is generated for every error)
• Other poorly configured or poorly designed programs/scripts that make a bunch of files and forget to delete them

Troubleshooting

If you find that you are low or out of inodes but don't know where most of the them are, I have this Bash one-liner that searches the current directory and counts the inodes in at a folder depth of 1:

for i in $(find $(pwd) -maxdepth 1 -type d | sort); do echo -e "$(find "$i" | wc -l)\t: $(readlink -f "$i")"; done | sort -nr

You can keep changing the current working directory starting from / until you find the culprit using up your inode allocation.

Explanation

Your VPS is on a VZFS filesystem, which is part of Parallels Virtuozzo (not OpenVZ, which is similar and based on the same technology, but OpenVZ wouldn't be using VZFS).

Due to the way Virtuozzo stores files in VZFS, inodes are often limited more so than they would be on other filesystems like ext4 or XFS. The host tracks all these files, and it would be advantageous to the hosting provider not to let a single VPS take up hundreds of millions of inodes. As a result, the hosting provider may set the inode limit to be low, like 1,000,000 inodes.

After years of working with hundreds of customers who exhausted their inode allocation on Virtuozzo, these "mysterious" disk quota issues don't surprise me anymore.

Cause: Other Virtuozzo Limits

A very small percentage of the Virtuozzo VPS customers I've worked with had filesystem issues because they hit other limits. You can see some (but not all) of the limits with this command:

cat /proc/user_beancounters

Troubleshooting

If the failcnt column has a value greater than 0 or a held column value is equal to the corresponding limit value, you have hit a limit.

You can look up what each parameter is on OpenVZ's wiki here. A parameter may be "primary", "secondary", or "auxiliary".

You should contact your hosting provider for further assistance if you find that you cannot decrease the held count for a limit that your VPS has reached.

This answer can be expanded a lot depending on which beans were maxed out, as different limits being reached cause different symptoms.

Cause: Limit(s) Decreased After Being Hit

Regarding /proc/user_beancounters or df -i, sometimes, a Virtuozzo system administrator may decrease the limit of a parameter below the held value.

For example, if the original limit of the diskinodes parameter was 1,500,000 and you hit the limit then someone at your hosting provider sets your inode limit to 1,000,000, you would see a bizarre inode report from df -i that makes no sense.

On your end, you could see an unreasonably large number, like 18,446,744,069,620,218,961.

I consider this to be a sinister behavior from the hosting provider, especially if they don't inform you, because the unusual values you see go against the knowledge of super users who don't have experience with Virtuozzo/OpenVZ, which leads to misleading advice (example, another example).

Troubleshooting

Contact your hosting provider. Show them what you found and work with them to get your held beans below the limit.

If they refuse to help you, ditch your hosting provider and find another one that doesn't use Virtuozzo/OpenVZ virtualization. KVM virtualization, VMware virtualization, Xen virtualization, or bare metal servers would be subject to far fewer limits than Virtuozzo/OpenVZ.

Explanation

Your hosting provider may have been auditing or responding to an alert and found that your VPS was using up too much of a specific resource (almost always the inodes limit, which is the diskinodes parameter on their end).

An inexperienced Virtuozzo admin at the hosting provider believes that they can cap the issue by reducing the limit to something lower than the actual resource usage. In the case of inodes, you may have a lower allocation, like 1,000,000, even though your actual current usage may be higher, like 1,500,000.

The Virtuozzo admin in their control panel would see your actual usage and the new limit, but you would see bogus numbers that are possibly very unreasonably high due to the way that Virtuozzo virtualizes.

A negligent Virtuozzo admin would not inform you of this change, which is why you should contact your hosting provider if this happens to you.

Answer 2

"Disk quota exceeded" doesn't mean that there is no space available on your disk, but that you are not allowed to use so much space.

In general, quotas are some limits set by administrator - maximum number of processes you can run, how much space you can occupy, how many files you can have, etc. Disk quotas can be set for maximum number of inodes and disk blocks.

Try running quota and quota -g (alternatively, you can run repquota -u and repqouta -g) to see if your user is restricted in this way and how. If yes, you can edit those quotas with edquota or turn them off with quotaoff command. You might need to log in as root for this, depending on your settings.

This site covers a few examples how to work with quotas and also contains links to manual pages to useful commands.