1

I have a Ubuntu docker run on CentOS host in Google Cloud. The docker runs a spring-boot application used for image uploading to S3. I do not save anything on local disk. Every 2 weeks or so, I have disk space issue (no space left on device). I run the following on the host machine:

[james@api /]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       1.0T   58G  967G   6% /
devtmpfs         15G     0   15G   0% /dev
tmpfs            15G     0   15G   0% /dev/shm
tmpfs            15G   26M   15G   1% /run
tmpfs            15G     0   15G   0% /sys/fs/cgroup
tmpfs           3.0G     0  3.0G   0% /run/user/0
tmpfs           3.0G     0  3.0G   0% /run/user/1000

[james@api /]$ sudo du -hs * | sort -rh | head -5
3.3G    var
1.5G    usr
154M    boot
58M     srv
34M     etc

I have run the same on the docker container:

[root@ea23811c1871 /]# df -h 
Filesystem      Size  Used Avail Use% Mounted on
overlay         1.0T   58G  966G   6% /
tmpfs            64M     0   64M   0% /dev
tmpfs            15G     0   15G   0% /sys/fs/cgroup
/dev/sda1       1.0T   58G  966G   6% /var/log
shm              64M     0   64M   0% /dev/shm
tmpfs            15G     0   15G   0% /proc/acpi
tmpfs            15G     0   15G   0% /proc/scsi
tmpfs            15G     0   15G   0% /sys/firmware

[root@ea23811c1871 /]# du -hs * | sort -rh | head -5
1.2G    usr
139M    var
2.7M    etc
36K     tmp
28K     root

My question is, how can I find where the 58G is hiding ??

Thanks, Hanan

Hanan Bareket
  • 13
  • 3

2 Answers2

0

First of all, if not haven't done this already, please setup monitors and alerts for the disk space usage. Nagios/Icinga has some generic plugins that are easy to setup or even a simple script that alerts you when the disk is becoming full would be helpful here.

Then you should check how much space your container is occupying via the docker commands (look into docker ps -s , docker system df or docker inspect -- read the man pages carefully before running these). You also need to consider spaces occupied by the configuration files user for the containers, checkpoint (if used), and log files. In generally poke around /var/lib/docker/ to get an idea on what files are residing where in the container.

Tux_DEV_NULL
  • 1,083
  • 7
  • 11
0

In host macchine run lsof and view deleted files

lsof |grep delete

Maybe the process did not release the deleted file

zersh
  • 151
  • 2
  • Well, this command returned 467770 open files !!!!!! good call!!! – Hanan Bareket Aug 28 '18 at 07:06
  • Is there an easy way to close these files without knowing the pid ?? – Hanan Bareket Aug 28 '18 at 07:13
  • use a combination of awk, sort and uniq for get pid. then pass the list to the kill command. be careful - you can kill the main process. it is better to first study the processes that you want to kill. – zersh Aug 29 '18 at 11:12
  • Thanks. However, I went for the approach of finding out why so many open files. Apparently, I have not closed input streams which I thought are closed automatically when the upload finishes. Will update later on .... – Hanan Bareket Aug 30 '18 at 12:04