git reports "No space left on device", but df -h says there is is 2.5G available

Question

I have an EC2 node running Ubuntu 14.04. On a deploy this morning, I received the following error message from git fetch:

error: unable to create temporary file: No space left on device

I logged into the server and df -h indicates I have plenty of space:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            492M   12K  492M   1% /dev
tmpfs           100M  488K   99M   1% /run
/dev/xvda1      7.8G  4.9G  2.5G  67% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none            497M  4.0K  497M   1% /run/shm
none            100M     0  100M   0% /run/user

Am I misreading df here? My understanding has been that /tmp on EC2 is resident on /dev/xvda1, but maybe I'm wrong?

I am currently researching what to do about node counts. Thanks so much guys. If I get a solution I will answer my own question, or feel free to answer it if you would like. — Rjak, Mar 16 '17 at 16:08
We have loads of disk full due to inode exhaustion Q&A already. — user9517, Mar 16 '17 at 16:35
@istheEnglishway Ok understood. Killing old kernels solved the problem for me. — Rjak, Mar 16 '17 at 17:00

score 6 · Accepted Answer · answered Mar 17 '17 at 02:48

6

Verify system inodes:

root #df -i /

If there is approaching 100% usage, try to using

xdiskusage /

To find out what is occupying the inodes.

There is, sadly enough, no way to increase the number of inodes on a file system once the file system has been created.

Except LVM, which can be expanding the number of inodes with resize2fs

Referred to:No space left on device while there is plenty of space available

answered Mar 17 '17 at 02:48

Edward Chan JW

176
3

What if I pathetically running out of space to install `xdiskusage` or `ncdu`? – tom10271 Oct 02 '21 at 04:44

score 0 · Answer 2 · answered Mar 17 '17 at 02:36

It could be some application is creating huge number of small files and completely exhausting the inodes. You may look for such rogue application and delete the unwanted files.
The inode limit can't be increased dynamically, however if you are using LVM you may think of increasing the size of the volume otherwise take a backup and create a new filesystem specifying higher inode limit.

score 0 · Answer 3 · edited Oct 14 '20 at 14:50

You can use find command to search for directory which has bulk files and you can perform clean up based on your standards

find /dev/xvda1 -type f -size +1M -exec ls -ltrh {} \; and when you find the place or directory where you need to perform the clean up pass that directory path in source and you can perform the clean up

find /dev/xvda1/dummy -type f -size +1M -mtime +120 -exec rm -rf  {} \;

In the above command I am removing the files which are older than 120 from a directory called dummy

git reports "No space left on device", but df -h says there is is 2.5G available

3 Answers3