0

I have a cloud server with rackspace. It is CentOS release 5.10 (Final). 40GB storage, 1gb ram, 2gb swap.

I am running the kloxo control panel to manage the virtual hosts.

Every day it eats up all of the ram and swap at 3:40am and then ultimately locks up entirely.

I have tried several commands to list all of the cron jobs such as 

for user in $(cut -f1 -d: /etc/passwd); do crontab -u $user -l; done

/var/spool/cron/*

/etc/crontab (and the daily, hour, weekly, monthy files)

I have a couple of php scripts that run about every 10 minutes, but nothing else is scheduled. I have disabled all of the log rotations, virus scan, and anything else that was scheduled.

I've been monitoring the processes during that time period using "top" and nothing stands out that is using a large amount of memory.

Network traffic is also minimal during that time period, so it doesn't seem to be any type of attack.

Any other time of the day when I run "top" I see minimal resources used

top - 14:39:21 up 10:14,  1 user,  load average: 0.08, 0.13, 0.09
Tasks:  92 total,   1 running,  91 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1048576k total,   995324k used,    53252k free,   154120k buffers
Swap:  2096472k total,        0k used,  2096472k free,   509536k cached

ClamAv and Maldet find no suspicious files

I am running out of ideas and am really trying to avoid rebuilding the server. Any advice on where else I could look for problems would be helpful.

In response to Peter H the results of /var/log/cron for that time period are:

May 29 03:10:01 shared-hosting crond[32243]: (root) CMD (/usr/lib64/sa/sa1 1 1)
May 29 03:20:01 shared-hosting crond[518]: (root) CMD (/usr/lib64/sa/sa1 1 1)
May 29 03:30:01 shared-hosting crond[1023]: (root) CMD (/usr/lib64/sa/sa1 1 1)
May 29 03:40:05 shared-hosting crond[2482]: (root) CMD (/usr/lib64/sa/sa1 1 1)
May 29 04:00:15 shared-hosting crond[2597]: (root) CMD (/usr/lib64/sa/sa1 1 1)
May 29 04:00:15 shared-hosting crond[2530]: (root) error: Job execution of per-minute job scheduled for 03:54 delayed into subsequent minute 04:00. Skipping job run.
May 29 04:00:17 shared-hosting crond[2530]: CRON (root) ERROR: cannot set security context
May 29 04:25:41 shared-hosting crond[1835]: (CRON) STARTUP (V5.0)
May 29 04:30:01 shared-hosting crond[2209]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Developer Gee
  • 101
  • 2
  • when does your logrotate run? – jhenn May 29 '15 at 18:45
  • You can check /var/log/cron it logs executions of jobs w timestamp. – Petter H May 29 '15 at 19:05
  • Petter H - I updated the question with the output of that log. It is my understanding that "/usr/lib64/sa/sa1 1 1/usr/lib64/sa/sa1 1 1" has to do with sysstat, which I installed yesterday. I was having the same issues before I isntalled sysstat. jhenn - The logs were rotating at 4am, but I removed that cron job a few days ago, because I thought it was causing the problem. – Developer Gee May 29 '15 at 19:23
  • I also restarted the server at 4:25 am, that is the reason for the "Startup" line – Developer Gee May 29 '15 at 19:23

1 Answers1

0

The control panel has a scheduled task that runs daily to clean up logs, record bandwidth usage, etc.

I haven't been able to find this anywhere in the cron, but I was able to disable it from the control panel. All is well now.

Developer Gee
  • 101
  • 2