5

The disk space on my Ubuntu web server, running on DigitalOcean, seems to be full. I have already found and deleted the large log file and restarted Nginx. However, it still seems like there is a problem.

df shows that it's still full:

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        50G   47G   22M 100% /

But du shows that it's not full:

$ du -shx /
3.2G    /

What do I need to do to fix this problem?

Andrew
  • 3,293
  • 8
  • 32
  • 35
  • I had something like that with Apache, not sure if it's the same with Nginx. The log files were written at offset, so deleting them while Apache was running regenerated them at the same size (i guess with zero data) and kept writting to them. Maybe stop nginx, delete the files and start it after that? – tilkov Apr 29 '20 at 18:56
  • I don't think it's the same issue because the large log file was not the nginx log but an application log. – Andrew Apr 29 '20 at 19:06
  • You can visually check where the data is located with `ncdu`. If you don't have it: `apt install ncdu` – tilkov Apr 29 '20 at 19:15
  • I just figured it out. Apparently `tail` was still referencing the deleted file. – Andrew Apr 29 '20 at 19:20

1 Answers1

6

First check and see if any processes are still referencing the deleted file:

$ sudo lsof / | grep deleted

In my case, it was showing that tail was still using it:

tail      18961       root    3r   REG  253,1 46511271936 1189657 /var/www/myapp/shared/log/production.log (deleted)

Running ps x and filtering by the process name showed me the process id that I needed to kill:

$ ps x | grep tail
 3376 pts/1    S+     0:00 grep --color=auto tail
18961 ?        Ss     0:00 tail -f /var/www/myapp/current/log/production.log

In this case, 18961 was the process id. Now kill the process:

kill 18961

Now df shows that there is free space again:

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        50G  3.3G   44G   7% /
Andrew
  • 3,293
  • 8
  • 32
  • 35