I have an issue almost identical to what is here: Cron jobs not working anymore
However I can't seem to find any memory-related problems as per the accepted answer, so I thought I should open a new question instead of answering on that one.
Versions:
uname -a
Linux __hostname__ 3.10.0-1062.1.1.el7.x86_64 #1 SMP Fri Sep 13 22:55:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
sudo systemctl --version
systemd 219
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN
cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)
My hourly cron jobs were executing successfully for a few weeks, but at some point they stopped executing on both of my servers (they didn't stop at the same time on both servers). The jobs are registered in the crontab of a technical user, i.e. no login shell for that user. They are set to execute at 03 minutes every hour (task is to import certain log files to DB and we accomodated 3 minutes for hourly log rollover).
Successful execution:
Nov 13 18:03:01 server02 CROND[8057]: (__tech_user__) CMD (/opt/xxx/LogImporter/start.sh &>/opt/xxx/LogImporter/import.log)
Nov 13 18:03:01 server02 CROND[8058]: (__tech_user__) CMD (/opt/yyy/LogImporter/start.sh &>/opt/yyy/LogImporter/import.log)
No other logs at that time. Unsuccessful execution an hour later:
Nov 13 19:03:01 server02 CROND[12006]: (__tech_user__) CMD (/opt/xxx/LogImporter/start.sh &>/opt/xxx/LogImporter/import.log)
Nov 13 19:03:01 server02 CROND[12007]: (__tech_user__) CMD (/opt/yyy/LogImporter/start.sh &>/opt/yyy/LogImporter/import.log)
Nov 13 19:03:01 server02 CROND[12009]: (CRON) EXEC FAILED (/usr/sbin/sendmail): Resource temporarily unavailable
Nov 13 19:03:01 server02 CROND[12008]: (CRON) EXEC FAILED (/usr/sbin/sendmail): Resource temporarily unavailable
Nov 13 19:03:01 server02 CROND[12002]: (__tech_user__) MAIL (mailed 71 bytes of output but got status 0x0001#012)
Nov 13 19:03:01 server02 CROND[12003]: (__tech_user__) MAIL (mailed 71 bytes of output but got status 0x0001#012)
Note: __tech_user__ is a replacement for actual username.
Server01 and Server02 are producing the same error logs, but Server01 "broke" 3 hours earlier on same day. When I launch the same script manually, it works fine. For reference, here is the start.sh:
#!/bin/bash
export JAVA_HOME=/opt/xxx/Java/jdk1.8.0_221
export PATH=$JAVA_HOME/bin:$PATH
#Use lockfile to avoid simultaneous instances
exec 200>/opt/xxx/LogImporter/start.lock
flock -n 200 || exit 1
java -cp /opt/xxx/LogImporter/LogImporter.jar x.y.z.logimporter.LogImporter /opt/xxx/LogImporter/
As I mentioned above, I don't see any clear memory issues. On Server01 I have double instance running of everything (separate environments), but there is still available memory:
top - 11:04:25 up 38 days, 22:43, 2 users, load average: 0.18, 0.25, 0.31
Tasks: 205 total, 1 running, 204 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.1 us, 0.3 sy, 0.0 ni, 98.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16265944 total, 321012 free, 7956004 used, 7988928 buff/cache
KiB Swap: 4063228 total, 3924988 free, 138240 used. 7148196 avail Mem
Server02 has even more (single environment only):
top - 12:01:17 up 38 days, 23:39, 3 users, load average: 0.23, 0.19, 0.21
Tasks: 199 total, 1 running, 198 sleeping, 0 stopped, 0 zombie
%Cpu(s): 4.0 us, 1.4 sy, 0.0 ni, 94.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16265944 total, 3711468 free, 5559448 used, 6995028 buff/cache
KiB Swap: 4063228 total, 3930364 free, 132864 used. 9602824 avail Mem
Interestingly, I have a daily cron job also on each server, running at 00:15 every day, and they also failed on same day (last successful run on Nov 13 00:15, failed since Nov 14 00:15 every day). My full crontab for the technical user:
Server01:
15 0 * * * /opt/xxx-env1/Logrotator/logrotate.sh &>/opt/xxx-env1/Logrotator/logrotate.log
15 0 * * * /opt/xxx-env2/Logrotator/logrotate.sh &>/opt/xxx-env2/Logrotator/logrotate.log
3 * * * * /opt/xxx-env1/LogImporter/start.sh &>/opt/xxx-env1/LogImporter/import.log
3 * * * * /opt/xxx-env2/LogImporter/start.sh &>/opt/xxx-env2/LogImporter/import.log
3 * * * * /opt/yyy/LogImporter/start.sh &>/opt/yyy/LogImporter/import.log
Server02:
15 0 * * * /opt/xxx/Logrotator/logrotate.sh &>/opt/xxx/Logrotator/logrotate.log
3 * * * * /opt/xxx/LogImporter/start.sh &>/opt/xxx/LogImporter/import.log
3 * * * * /opt/yyy/LogImporter/start.sh &>/opt/yyy/LogImporter/import.log
I tried to comment out some jobs, and leaving only 1 in each server, but still the same issue. What I also noticed, and I'm not sure if it is a problem or not, is that crond.service shows the TasksCurrent is very high:
sudo systemctl show crond.service
Type=simple
Restart=on-failure
NotifyAccess=none
RestartUSec=30s
...
MemoryCurrent=18446744073709551615
TasksCurrent=18446744073709551615
Delegate=no
CPUAccounting=no
CPUShares=18446744073709551615
StartupCPUShares=18446744073709551615
CPUQuotaPerSecUSec=infinity
...
MemoryAccounting=no
MemoryLimit=18446744073709551615
DevicePolicy=auto
TasksAccounting=no
TasksMax=18446744073709551615
However, I see the same numbers on my test server, and crontab is executing fine (although I do not have the hourly jobs, only the daily jobs).
