0

The servers at my work have a script made by me that every midnight copies log files to a folder, copies this folder to Amazon S3 and zeroes the original files.

However, I saw that the original files haven't been zeroed and this is affecting disk space.

The command I use is the following:

cat /dev/null | tee /etc/vinter/logs/*

Today I tried using this command separately and it didn't work. After that, tried other two and none worked.

[root@APPSERVER11-S1 ~]# cat /dev/null | tee /etc/vinter/logs/*
[root@APPSERVER11-S1 ~]# ls -l /etc/vinter/logs/*
-rw-r--r-- 1 root root 9223905280 Sep 11 12:47 /etc/vinter/logs/rsp.appserver11-s1.apigoldnode-1.rsp.log
-rw-r--r-- 1 root root    9468202 Sep 11 12:47 /etc/vinter/logs/Services-APIGOLDNODE-1.log
-rw-r--r-- 1 root root          0 Sep 11 12:47 /etc/vinter/logs/Services-APIGOLDNODE-1.log.1
[root@APPSERVER11-S1 ~]#
[root@APPSERVER11-S1 ~]#
[root@APPSERVER11-S1 ~]# echo -n | tee /etc/vinter/logs/*
[root@APPSERVER11-S1 ~]# ls -l /etc/vinter/logs/*
-rw-r--r-- 1 root root 9224224768 Sep 11 12:48 /etc/vinter/logs/rsp.appserver11-s1.apigoldnode-1.rsp.log
-rw-r--r-- 1 root root    9493203 Sep 11 12:48 /etc/vinter/logs/Services-APIGOLDNODE-1.log
-rw-r--r-- 1 root root          0 Sep 11 12:47 /etc/vinter/logs/Services-APIGOLDNODE-1.log.1
[root@APPSERVER11-S1 ~]#
[root@APPSERVER11-S1 ~]#
[root@APPSERVER11-S1 ~]# printf '' | tee /etc/vinter/logs/*
[root@APPSERVER11-S1 ~]# ls -l /etc/vinter/logs/*
-rw-r--r-- 1 root root 9224380416 Sep 11 12:48 /etc/vinter/logs/rsp.appserver11-s1.apigoldnode-1.rsp.log
-rw-r--r-- 1 root root    9502868 Sep 11 12:48 /etc/vinter/logs/Services-APIGOLDNODE-1.log
-rw-r--r-- 1 root root          0 Sep 11 12:48 /etc/vinter/logs/Services-APIGOLDNODE-1.log.1

It's not working individually as well:

[root@APPSERVER11-S1 ~]# echo > /etc/vinter/logs/rsp.appserver11-s1.apigoldnode-1.rsp.log
[root@APPSERVER11-S1 ~]# ls -l /etc/vinter/logs/*
-rw-r--r-- 1 root root 9236357120 Sep 11 12:58 /etc/vinter/logs/rsp.appserver11-s1.apigoldnode-1.rsp.log
-rw-r--r-- 1 root root     520807 Sep 11 12:58 /etc/vinter/logs/Services-APIGOLDNODE-1.log
-rw-r--r-- 1 root root   10264642 Sep 11 12:54 /etc/vinter/logs/Services-APIGOLDNODE-1.log.1

The truncate command, as instructed in the comments, didn't work as well:

[root@APPSERVER11-S1 ~]# truncate -s 0 /etc/vinter/logs/rsp.appserver11-s1.apigoldnode-1.rsp.log
[root@APPSERVER11-S1 ~]# ls -l /etc/vinter/logs/
total 8292
-rw-r--r-- 1 root root 9283125248 Sep 11 13:32 rsp.appserver11-s1.apigoldnode-1.rsp.log
-rw-r--r-- 1 root root    6026198 Sep 11 13:32 Services-APIGOLDNODE-1.log
-rw-r--r-- 1 root root          0 Sep 11 13:06 Services-APIGOLDNODE-1.log.1

The tee version is:

[root@APPSERVER11-S1 ~]# tee --version
tee (GNU coreutils) 8.22
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Mike Parker, Richard M. Stallman, and David MacKenzie.

The filesystem is xfs.

[root@APPSERVER11-S1 ~]# df -Th
Filesystem                     Type      Size  Used Avail Use% Mounted on
devtmpfs                       devtmpfs  7.3G     0  7.3G   0% /dev
tmpfs                          tmpfs     7.3G     0  7.3G   0% /dev/shm
tmpfs                          tmpfs     7.3G  185M  7.1G   3% /run
tmpfs                          tmpfs     7.3G     0  7.3G   0% /sys/fs/cgroup
/dev/mapper/vg_main-lv_root    xfs        16G   14G  1.8G  89% /
/dev/mapper/vg_dados-lv_dados  xfs        20G   33M   20G   1% /datastorage
/dev/mapper/vg_dados-lv_docker btrfs      80G  128M   79G   1% /var/lib/docker
/dev/xvdb1                     xfs       497M  161M  337M  33% /boot
tmpfs                          tmpfs     1.5G     0  1.5G   0% /run/user/0
tmpfs                          tmpfs     1.5G     0  1.5G   0% /run/user/1002

I tried in another server and only the most constantly updated file (a lot of lines each second) didn't get zeroed. Could it be that the file is so constantly updated it can't even get zeroed?

[root@APPSERVER21-S1 ~]# ls -l /etc/vinter/logs/
total 2046352
-rw-r--r-- 1 root root     3920496 Sep 11 14:35 PlusoftCRMIntegration-APISILVERNODE-1.log
-rw-r--r-- 1 root root 30891423129 Sep 11 15:27 rsp.appserver21-s1.apigoldnode-1.api-oi-gold.log
-rw-r--r-- 1 root root   406151168 Sep 11 15:27 rsp.appserver21-s1.apisilvernode-1.api-oi-silver.log
[root@APPSERVER21-S1 ~]# cat /dev/null | tee /etc/vinter/logs/*
[root@APPSERVER21-S1 ~]# ls -l /etc/vinter/logs/
total 492
-rw-r--r-- 1 root root           0 Sep 11 15:28 PlusoftCRMIntegration-APISILVERNODE-1.log
-rw-r--r-- 1 root root 30892328752 Sep 11 15:28 rsp.appserver21-s1.apigoldnode-1.api-oi-gold.log
-rw-r--r-- 1 root root           0 Sep 11 15:28 rsp.appserver21-s1.apisilvernode-1.api-oi-silver.log

This second server's /partition filesystem is xfs, like the first one.

[root@APPSERVER21-S1 ~]# df -Th
Filesystem                     Type      Size  Used Avail Use% Mounted on
devtmpfs                       devtmpfs  3.6G     0  3.6G   0% /dev
tmpfs                          tmpfs     3.6G     0  3.6G   0% /dev/shm
tmpfs                          tmpfs     3.6G  137M  3.4G   4% /run
tmpfs                          tmpfs     3.6G     0  3.6G   0% /sys/fs/cgroup
/dev/mapper/vg_main-lv_root    xfs        16G  3.0G   13G  20% /
/dev/mapper/vg_dados-lv_dados  xfs        20G   33M   20G   1% /datastorage
/dev/mapper/vg_dados-lv_docker btrfs      80G  128M   79G   1% /var/lib/docker
/dev/xvdb1                     xfs       497M  161M  337M  33% /boot
tmpfs                          tmpfs     722M     0  722M   0% /run/user/0

Sometimes some of the log files get zeroed but return to their original size a few seconds later.

[root@APPSERVER11-S1 ~]# cat /dev/null | tee /etc/vinter/logs/*
[root@APPSERVER11-S1 ~]# ls -l /etc/vinter/logs/
total 24
-rw-r--r-- 1 root root       0 Sep 11 18:13 rsp.appserver11-s1.apigoldnode-1.rsp.log
-rw-r--r-- 1 root root 3125765 Sep 11 18:13 Services-APIGOLDNODE-1.log
-rw-r--r-- 1 root root       0 Sep 11 18:13 Services-APIGOLDNODE-1.log.1
[root@APPSERVER11-S1 ~]# ls -l /etc/vinter/logs/
total 160
-rw-r--r-- 1 root root 9618808832 Sep 11 18:13 rsp.appserver11-s1.apigoldnode-1.rsp.log
-rw-r--r-- 1 root root    3131397 Sep 11 18:13 Services-APIGOLDNODE-1.log
-rw-r--r-- 1 root root          0 Sep 11 18:13 Services-APIGOLDNODE-1.log.1

What can be going wrong?

JM Calil
  • 1
  • 3
  • See: https://serverfault.com/questions/221337/logrotate-successful-original-file-goes-back-to-original-size/221343 for explanation. TL;DR - application needs to cooperate with regards to log-rotation. If the filesystem supports sparse files, the "length of the file" and "size of the file" isn't necessarily the same, so the file truncation might actually free disk space, despite ls saying that the length of the file is still a lot. – Kjetil Joergensen Sep 11 '18 at 22:27
  • It's not just a matter of freeing disk space. I want the information in the archived log files to be found as easily as possible by the devs, which means the logs should rotate correctly so they have only the information for the current day. It's not the ideal scenario if they can't be zeroed. I used `ls -ls` after trying to empty one of the log files and the leftmost column showed the number 108. What does it mean? – JM Calil Sep 11 '18 at 22:50
  • ls -s gives you the number of blocks used, you need to multiply this number by blocksize. Regardless - by the sounds of it - what you want is either for the application itself to rotate the logfiles at specific intervals, or, the application needs to support cooperative logrotation (some other process, i.e. logrotate, renames the logfile, and signals the application to re-open the logfile). If the application logs to stdout/stderr, there's "helper applications" like cronolog as well. – Kjetil Joergensen Sep 11 '18 at 22:57
  • There's no logrotate configured for the problematic folder (/etc/vinter/logs). I want the mentioned script (which is separated from the application) to be able to empty the logfiles when the time specified in crontab comes. – JM Calil Sep 11 '18 at 23:17
  • In order to do that you *need* to cooperate with the application (see i.e. the design of logrotate, move the file, signal the application to re-open the file). What happens when you "empty the file" is, it gets truncated to zero length. The application still holds this file open though, the offset into the file remains, and when the application logs to the file again, it'll do so at that offset. So, in effect you end up with a file of "all 0x00 bytes up to where the applications file descriptors offset is". – Kjetil Joergensen Sep 12 '18 at 17:52

1 Answers1

0

Have you tried to delete the whole file forcefully by running?

# rm /var/log/logfile -f
Hannan
  • 19
  • 1
  • 2
    I can't. If I delete the log files from their original folder, they will only reappear if I restart the application, which I can't do. – JM Calil Sep 12 '18 at 01:03