1

I have a low memory VPS that mysql and the swap file crash out every morning at exactly 0300. There are no cronjobs on the system that have been configured. The server is a basic LAMP development server and all settings are defaults.

 cat /var/log/messages|grep -i mysql
Jun 25 20:51:07 vader sshd[72946]: error: PAM: authentication error for mysql from 65.52.155.140
Jun 28 03:01:34 vader kernel: pid 848 (mysqld), uid 88, was killed: out of swap space
Jun 28 03:01:34 vader kernel: pid 93947 (mysqld), uid 88, was killed: out of swap space
Jun 29 03:01:32 vader kernel: pid 98578 (mysqld), uid 88, was killed: out of swap space
Jun 29 03:01:33 vader kernel: pid 2586 (mysqld), uid 88, was killed: out of swap space

My swap file is 1 gig. I tried 2 gigs, the same pattern of crashing begins after a week.

 ls -l /home/sw*
-rw-------  1 root  wheel  1073741824 Jun 21 13:19 /home/swap0

Even worse, I can't re-initialize the swap file without a reboot

swapoff -a mdconfig -a -t vnode -f /home/swap0 -u 0 && swapon /dev/md0 mdconfig: ioctl(/dev/mdctl): Device busy

I don't have much memory to work with:

# vmstat
 procs      memory      page                    disks     faults         cpu
 r b w     avm    fre   flt  re  pi  po    fr  sr vt0 md0   in   sy   cs us sy id
 1 0 3   1709M   491M    44   0   0   0    54  29   0   0    4  135   94  0  0 100

But I shouldn't need it if mysql was running correctly.

Two questions.

no. 1) How do I reinitialize a swap file after it crashes so I don't have to reboot (I would just like to know because everything I find on google fails)?

no. 2) How do I stabilize mysql so that it doesn't burp at 0300 for massive amounts of memory?

Deer Hunter
  • 1,070
  • 7
  • 17
  • 25
NIX
  • 11
  • 3
  • 2
    1. There's a scheduled task *somewhere*. 2. You can't, your system ran out of memory and it's in an unstable state *at best*. Reboot. 3. Maybe you should get more RAM added to your VPS. – Sammitch Jun 29 '15 at 18:15
  • This issue still exist removing the only error in logs, a DHCP query. After removing DHCP and setting a static IP, same issue. :( – NIX Aug 09 '15 at 01:16

1 Answers1

0

I was just seeing this behaviour as well, except in my case the mysql server was dying on a weekly basis instead of a daily basis. It happened at the same time as the weekly periodic tasks run (see /etc/crontab and /etc/cron.d/periodic/{daily,weekly}/). From your log it looks like it happens at 03:01 which is during the daily periodic task run (starts at 03:00). There are a few possible solutions:

  • Avoid running the periodic tasks, or at least identify which task is eating up memory. In my case it was the makewhatis command that runs weekly
  • Add more swap. Past a certain point there should be enough memory for the server to coexist with the periodic tasks
  • Add another periodic task that runs after the others and restarts the server
staktrace
  • 101