Errno::ENOSPC: No space left on device in Ruby on Rails

Question

Iam hosting a Ruby on Rails Application on a Linode with 96 GB storage. All the images of my application are stored on amazon and they are fetched from amazon. Thus my linode should have a minimum of 80 GB storage free.

But when the log files of the application exceed 200 Mb I get the error:

Errno::ENOSPC: No space left on device

and my website goes down.

I enabled log rotation for my application on daily basis with the following code in file /etc/logrotate.conf:

path/to/application/log/*.log {
  daily
  missingok
  rotate 1
  compress
  delaycompress
  notifempty
  copytruncate
}

But now I have started getting the same error twics a day so I changed my config settings on the basis of size as:

path/to/application/log/*.log {
  size 1M
  missingok
  rotate 1
  compress
  delaycompress
  notifempty
  copytruncate
}

I also moved the logrotate file from /etc/cron.daily to /etc/cron.hourly so that the logrotate cronjob runs each hour and rotate the log if it exceeds 1M. I restarted the apache.

But when I checked after an hour my files sizes were greater than 1M. Thus the logrotate cronjob didnt work after every hour. When I forcefully rotate the log using the command:

 sudo /usr/sbin/logrotate -f /etc/logrotate.conf

then the log files gets rotated and the files size decrease.

I am not able to figure out why this configuration doesnt work. Moreover when the space on linode is 80 GB how does it affect if log files exceeds 200 Mb. What is the way to allocate more space to log directory?

As suggested I ran df -ih to check if Iam running out of inodes. The follwoing output I get:

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/xvda               2.2M    263K    2.0M   12% /
devtmpfs                174K    2.0K    172K    2% /dev
none                    174K       1    174K    1% /dev/shm
none                    174K      33    174K    1% /var/run
none                    174K       2    174K    1% /var/lock
none                    174K       1    174K    1% /lib/init/rw

Thus my inodes are almost free. Iam not running out of inodes.

The output of df -h is as:

Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda             9.5G  8.7G  325M  97% /
devtmpfs             1010M  112K 1010M   1% /dev
none                 1010M     0 1010M   0% /dev/shm
none                 1010M   52K 1010M   1% /var/run
none                 1010M     0 1010M   0% /var/lock
none                 1010M     0 1010M   0% /lib/init/rw

My disk space is really less. How should I proceed?

Please help.Many Thanks!

Answer 1

I had a the same error message when trying to run an npm install.

npm ERR! nospc ENOSPC: no space left on device, open /{dir}/...

The fix for me was the following:

• sudo rm -rf node_modules - Remove node_modules
• df -hi - Check the available disk space
• sudo apt-get update - Update packages
• sudo apt-get autoremove - Use 'autoremove' to remove packages that are no longer needed
• sudo apt-get -f install - Correct broken dependencies

Answer 2

It's true, you have very little space left.

Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda             9.5G  8.7G  325M  97% /

Only a few hundred megabytes, and your disk will be full again.

You said you expected to have much more disk space than this shows, so I would recommend you contact Linode to find out what's going on.

Answer 3

my 2 cents here:

I found out that the output of lsof | grep -i deleted was having production.log from several ages ago. Which a restart of the service freed up all the space. I got the idea from: Disk full, du tells different. How to further investigate?