Ubuntu Mysql stop working after outofmemory in a clamscan, mysqld.sock

Question

I'm having daily problems with postman with autentification througth mysql on my Ubuntu14LTS Server. I think the problem arrise after a clamscan scron that breaks in an Out of Memory. In some way mysql socket gets break and I can't read my emails from an email client due to authdaemond auth error, but I can use mysql databases via mysql command, and also my webapp works ok with the same mysql server. If I restart mysql service, authdaemond works.

How could I fix this? any advice will be appreciate, thanks.

error (log)

05:06:42 kernel: [449190.488344] Out of memory in UB 56629: OOM killed process 7839 (clamscan) score 0 vm:304480kB, rss:243064kB, swap:0kB
05:06:42 amavis[32693]: (32693-04) (!!)AV: ALL VIRUS SCANNERS FAILED
05:11:15 kernel: [449463.682629] Out of memory in UB 56629: OOM killed process 15307 (clamscan) score 0 vm:362608kB, rss:294436kB, swap:0kB
05:11:15 amavis[32693]: (32693-05-6) (!!)AV: ALL VIRUS SCANNERS FAILED
05:11:53 kernel: [449501.302589] Out of memory in UB 56629: OOM killed process 16511 (clamscan) score 0 vm:374780kB, rss:306480kB, swap:0kB
05:11:53 amavis[32693]: (32693-05-7) (!!)AV: ALL VIRUS SCANNERS FAILED
05:13:43 kernel: [449611.701346] Out of memory in UB 56629: OOM killed process 20750 (clamscan) score 0 vm:377420kB, rss:309408kB, swap:0kB
05:13:43 amavis[32693]: (32693-06-2) (!!)AV: ALL VIRUS SCANNERS FAILED
05:15:27 kernel: [449715.323649] Out of memory in UB 56629: OOM killed process 23783 (clamscan) score 0 vm:374784kB, rss:306516kB, swap:0kB
05:15:27 amavis[32693]: (32693-06-4) (!!)AV: ALL VIRUS SCANNERS FAILED
05:17:36 kernel: [449844.487208] Out of memory in UB 56629: OOM killed process 24521 (clamscan) score 0 vm:378464kB, rss:310460kB, swap:0kB
05:17:36 amavis[32693]: (32693-06-6) (!!)AV: ALL VIRUS SCANNERS FAILED
05:22:09 kernel: [450118.059783] Out of memory in UB 56629: OOM killed process 25385 (clamscan) score 0 vm:380276kB, rss:312244kB, swap:0kB
05:22:09 amavis[32693]: (32693-07-7) (!!)AV: ALL VIRUS SCANNERS FAILED
05:22:39 kernel: [450147.625891] Out of memory in UB 56629: OOM killed process 25389 (clamscan) score 0 vm:378472kB, rss:310460kB, swap:0kB
05:22:39 amavis[32693]: (32693-07-8) (!!)AV: ALL VIRUS SCANNERS FAILED

09:12:59 imapd-ssl: Connection, ip=[::ffff:xx.xx.xx.xx]
09:13:00 authdaemond: failed to connect to mysql server (server=localhost, userid=mail): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
09:13:00 imapd-ssl: LOGIN FAILED, method=PLAIN, ip=[::ffff:xx.xx.xx.xx]
09:13:00 imapd-ssl: authentication error: Input/output error

Memory:

             total       used       free     shared    buffers     cached
Mem:          1024        967         56         67          0        292
-/+ buffers/cache:        674        349
Swap:          128        126          1

Process:

UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 sep24 ?        00:00:01 init
root         2     1  0 sep24 ?        00:00:00 [kthreadd/56629]
root         3     2  0 sep24 ?        00:00:00 [khelper/56629]
root         4     2  0 sep24 ?        00:00:00 [rpciod/56629/0]
root         5     2  0 sep24 ?        00:00:00 [rpciod/56629/1]
root         6     2  0 sep24 ?        00:00:00 [rpciod/56629/2]
root         7     2  0 sep24 ?        00:00:00 [rpciod/56629/3]
root         8     2  0 sep24 ?        00:00:00 [rpciod/56629/4]
root         9     2  0 sep24 ?        00:00:00 [rpciod/56629/5]
root        10     2  0 sep24 ?        00:00:00 [rpciod/56629/6]
root        11     2  0 sep24 ?        00:00:00 [rpciod/56629/7]
root        12     2  0 sep24 ?        00:00:00 [nfsiod/56629]
message+   221     1  0 sep24 ?        00:00:00 dbus-daemon --system --fork
root       249     1  0 sep24 ?        00:00:00 /lib/systemd/systemd-logind
root       252     1  0 sep24 ?        00:00:00 upstart-socket-bridge --daemon
root       253     1  0 sep24 ?        00:00:00 upstart-file-bridge --daemon
root       485     1  0 sep24 ?        00:00:00 /usr/sbin/xinetd -dontfork -pidfile /var/run/xinetd.pid -stayalive -inetd_compat -inetd_ipv6
root       491     1  0 sep24 ?        00:00:00 /usr/sbin/dovecot -F -c /etc/dovecot/dovecot.conf
root       496     1  0 sep24 ?        00:00:00 /usr/sbin/sshd -D
root       498     1  0 sep24 ?        00:00:03 cron
dovecot    541   491  0 sep24 ?        00:00:00 dovecot/anvil
root       542   491  0 sep24 ?        00:00:00 dovecot/log
root       545   491  0 sep24 ?        00:00:00 dovecot/config
syslog     563     1  0 sep24 ?        00:00:01 /sbin/syslogd -u syslog
root       611     1  0 sep24 ?        00:00:00 supervising syslog-ng                                  
root       612   611  0 sep24 ?        00:00:04 /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid --no-caps
amavis     757     1  0 sep24 ?        00:00:08 /usr/sbin/amavisd-new (master)
root       762     1  0 sep24 ?        00:00:46 /usr/sbin/spamd --create-prefs --max-children 5 --helper-home-dir -d --pidfile=/var/run/spamd.pid
root       770   762  0 sep24 ?        00:00:00 spamd child
root       772   762  0 sep24 ?        00:00:00 spamd child
bind       828     1  0 sep24 ?        00:00:00 /usr/sbin/named -u bind
clamav    1199     1  0 sep24 ?        00:00:35 /usr/sbin/clamd
clamav    1323     1  0 sep24 ?        00:02:30 /usr/bin/freshclam -d --quiet
root      1348     1  0 sep24 ?        00:00:00 /usr/sbin/courierlogger -pid=/var/run/courier/authdaemon/pid -start /usr/lib/courier/courier-authlib/authdaemond
root      1349  1348  0 sep24 ?        00:00:00 /usr/lib/courier/courier-authlib/authdaemond
root      1374  1349  0 sep24 ?        00:00:00 /usr/lib/courier/courier-authlib/authdaemond
root      1375  1349  0 sep24 ?        00:00:00 /usr/lib/courier/courier-authlib/authdaemond
root      1376  1349  0 sep24 ?        00:00:00 /usr/lib/courier/courier-authlib/authdaemond
root      1377  1349  0 sep24 ?        00:00:00 /usr/lib/courier/courier-authlib/authdaemond
root      1378  1349  0 sep24 ?        00:00:00 /usr/lib/courier/courier-authlib/authdaemond
root      1381     1  0 sep24 ?        00:00:00 /usr/sbin/courierlogger -pid=/var/run/courier/imapd.pid -start -name=imapd /usr/sbin/couriertcpd -address=0 -maxprocs=40 -maxperip=20 -nodnslookup -noi
root      1382  1381  0 sep24 ?        00:00:00 /usr/sbin/couriertcpd -address=0 -maxprocs=40 -maxperip=20 -nodnslookup -noidentlookup 143 /usr/lib/courier/courier/imaplogin /usr/bin/imapd Maildir
root      1411     1  0 sep24 ?        00:00:00 /usr/sbin/courierlogger -pid=/var/run/courier/imapd-ssl.pid -start -name=imapd-ssl /usr/sbin/couriertcpd -address=0 -maxprocs=40 -maxperip=20 -nodnsloo
root      1412  1411  0 sep24 ?        00:00:00 /usr/sbin/couriertcpd -address=0 -maxprocs=40 -maxperip=20 -nodnslookup -noidentlookup 993 /usr/bin/couriertls -server -tcpd /usr/lib/courier/courier/i
root      1434     1  0 sep24 ?        00:00:00 /usr/sbin/courierlogger -pid=/var/run/courier/pop3d.pid -start -name=pop3d /usr/sbin/couriertcpd -maxprocs=40 -maxperip=4 -nodnslookup -noidentlookup -
root      1435  1434  0 sep24 ?        00:00:00 /usr/sbin/couriertcpd -maxprocs=40 -maxperip=4 -nodnslookup -noidentlookup -address=0 110 /usr/lib/courier/courier/courierpop3login /usr/lib/courier/co
root      1463     1  0 sep24 ?        00:00:00 /usr/sbin/courierlogger -pid=/var/run/courier/pop3d-ssl.pid -start -name=pop3d-ssl /usr/sbin/couriertcpd -address=0 -maxprocs=40 -maxperip=4 -nodnslook
root      1464  1463  0 sep24 ?        00:00:00 /usr/sbin/couriertcpd -address=0 -maxprocs=40 -maxperip=4 -nodnslookup -noidentlookup 995 /usr/bin/couriertls -server -tcpd /usr/lib/courier/courier/co
root      2004     1  0 sep24 ?        00:00:04 /usr/lib/postfix/master
postfix   2009  2004  0 sep24 ?        00:00:02 qmgr -l -t fifo -u
root      2044     1  0 sep24 ?        00:00:00 /usr/sbin/saslauthd -a pam -r -c -m /var/spool/postfix/var/run/saslauthd -n 5
root      2045  2044  0 sep24 ?        00:00:00 /usr/sbin/saslauthd -a pam -r -c -m /var/spool/postfix/var/run/saslauthd -n 5
root      2046  2044  0 sep24 ?        00:00:00 /usr/sbin/saslauthd -a pam -r -c -m /var/spool/postfix/var/run/saslauthd -n 5
root      2047  2044  0 sep24 ?        00:00:00 /usr/sbin/saslauthd -a pam -r -c -m /var/spool/postfix/var/run/saslauthd -n 5
root      2048  2044  0 sep24 ?        00:00:00 /usr/sbin/saslauthd -a pam -r -c -m /var/spool/postfix/var/run/saslauthd -n 5
root      2177     1  0 sep24 tty1     00:00:00 /sbin/getty 38400 console
root      2179     1  0 sep24 tty2     00:00:00 /sbin/getty 38400 tty2
root      2180     1  0 sep24 ?        00:00:00 /sbin/getty -8 38400 tty1
mysql     7612     1  0 12:27 ?        00:01:12 /usr/sbin/mysqld
tomcat7   7703     1  1 12:27 ?        00:03:31 /usr/lib/jvm/default-java/bin/java -Djava.util.logging.config.file=/var/lib/tomcat7/conf/logging.properties -Djava.util.logging.manager=org.apache.juli
root      7812     1  0 12:27 ?        00:00:03 /usr/sbin/apache2 -k start
www-data  7815  7812  0 12:27 ?        00:00:06 /usr/sbin/apache2 -k start
www-data  7816  7812  0 12:27 ?        00:00:06 /usr/sbin/apache2 -k start
www-data  7819  7812  0 12:27 ?        00:00:06 /usr/sbin/apache2 -k start
www-data  7820  7812  0 12:27 ?        00:00:05 /usr/sbin/apache2 -k start
www-data  7821  7812  0 12:27 ?        00:00:05 /usr/sbin/apache2 -k start
www-data  7912  7812  0 12:30 ?        00:00:06 /usr/sbin/apache2 -k start
postfix  16508  2004  0 sep24 ?        00:00:00 tlsmgr -l -t unix -u -c
amavis   24919   757  0 12:40 ?        00:00:02 /usr/sbin/amavisd-new (ch1-avail)
amavis   24921   757  0 12:40 ?        00:00:03 /usr/sbin/amavisd-new (ch1-avail)
www-data 25335  7812  0 13:53 ?        00:00:03 /usr/sbin/apache2 -k start
www-data 25337  7812  0 13:53 ?        00:00:04 /usr/sbin/apache2 -k start
www-data 25338  7812  0 13:53 ?        00:00:03 /usr/sbin/apache2 -k start
postfix  26111  2004  0 16:40 ?        00:00:00 pickup -l -t fifo -u -c -o content_filter= -o receive_override_options=no_header_body_checks
root     26151   496  0 16:53 ?        00:00:00 sshd: root@pts/1    
root     26202 26151  0 16:54 pts/1    00:00:00 -bash
postfix  26237  2004  0 16:55 ?        00:00:00 cleanup -z -t unix -u -c
postfix  26238  2004  0 16:55 ?        00:00:00 trivial-rewrite -n rewrite -t unix -u -c
postfix  26241  2004  0 16:55 ?        00:00:00 smtp -t unix -u -c
postfix  26251  2004  0 16:55 ?        00:00:00 bounce -z -t unix -u -c
postfix  26252  2004  0 16:55 ?        00:00:00 bounce -z -t unix -u -c
root     26280 26202  0 16:56 pts/1    00:00:00 ps -ef

What kind of system is this? How much memory does it have ? How much swap does it have? What other relevant messages are logged in the various system logs? — user9517, Sep 25 '15 at 10:25
Its an Ubuntu14.10LTS server in a VPS, I've updated memory in question, and also put a list of process running. I didn't any other relevant messages on logs. — surfealokesea, Sep 25 '15 at 15:01

score 1 · Answer 1 · edited Apr 13 '17 at 12:13

1

Basically your system is running out of memory. What exctly is causing this should be in your log files somewhere. Have a look at this excellent Q&A to see the kind of information you are looking for.

Right now your solutions are to

add more swap
Increase memory size (move to a bigger instance).
Diagnose the problem and tune memory usage if possible (seems unlikely de to lack of logs and skillset).

edited Apr 13 '17 at 12:13

Community

1

answered Sep 25 '15 at 16:53

user9517

114,104
20
206
289

It seems as amavis is making a full scan, isn't it? I don't think i cant change swap, neither i can/want to upgrade server. Can I configure amavis to scan in a different way or limit memory usage? My crons are running at 5am. It seems that amavis too. Can I change amavis full scan hour? thanks Iain – surfealokesea Sep 25 '15 at 16:58

Ubuntu Mysql stop working after outofmemory in a clamscan, mysqld.sock

1 Answers1