1
I have an ec2 server and sometimes I have memory issues - I've had the mysql process killed due to out of memory. This breaks my website and game and sometimes it takes me hours or days to notice :(. Right now I have lamp, wordpress, and my java mmorpg running on the server. I know my mmorpg server takes quite a bit of RAM to hold all of the entities and data.
I have 1 GB of RAM and 1 GB of swap. Here is my top output currently.
What I don't understand is why if my java process is 7.6% and my current mysqld is 1.9% how I am even close? It says 956MB of Mem used and 378MB of Swap used - those numbers seem real. How is that even close to 7.6 + 1.9%?
Is there some way I can monitor my memory a bit better or something that I can configure on mysql, so it does not run OOM? I am pretty sure there is no leak in my java server code - but not 100% sure.
Sorry for all of the questions - after coding my mmorpg from scratch for over 5 years now I want to make sure my server is ready to handle a decent playerbase.
EDIT requested in comments
free output
total used free shared buffers cached
Mem: 1019336 937316 82020 344 8920 45120
-/+ buffers/cache: 883276 136060
Swap: 1048572 417708 630864
ps aux output
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 19644 0 ? Ss Aug14 0:00 /sbin/init
root 2 0.0 0.0 0 0 ? S Aug14 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S Aug14 0:08 [ksoftirqd/0]
root 4 0.0 0.0 0 0 ? S Aug14 0:00 [kworker/0:0]
root 5 0.0 0.0 0 0 ? S< Aug14 0:00 [kworker/0:0H]
root 7 0.0 0.0 0 0 ? S Aug14 1:51 [rcu_sched]
root 8 0.0 0.0 0 0 ? S Aug14 0:00 [rcu_bh]
root 9 0.0 0.0 0 0 ? S Aug14 0:00 [migration/0]
root 10 0.0 0.0 0 0 ? S Aug14 0:00 [kdevtmpfs]
root 11 0.0 0.0 0 0 ? S< Aug14 0:00 [netns]
root 12 0.0 0.0 0 0 ? S< Aug14 0:00 [perf]
root 15 0.0 0.0 0 0 ? S Aug14 0:00 [xenwatch]
root 20 0.0 0.0 0 0 ? S Aug14 0:00 [xenbus]
root 21 0.0 0.0 0 0 ? S Aug14 0:19 [kworker/0:1]
root 128 0.0 0.0 0 0 ? S Aug14 0:00 [khungtaskd]
root 129 0.0 0.0 0 0 ? S< Aug14 0:00 [writeback]
root 131 0.0 0.0 0 0 ? SN Aug14 0:00 [ksmd]
root 132 0.0 0.0 0 0 ? SN Aug14 0:00 [khugepaged]
root 133 0.0 0.0 0 0 ? S< Aug14 0:00 [crypto]
root 134 0.0 0.0 0 0 ? S< Aug14 0:00 [kintegrityd]
root 135 0.0 0.0 0 0 ? S< Aug14 0:00 [bioset]
root 137 0.0 0.0 0 0 ? S< Aug14 0:00 [kblockd]
root 488 0.0 0.0 0 0 ? S< Aug14 0:00 [md]
root 616 0.0 0.0 0 0 ? S Aug14 0:46 [kswapd0]
root 617 0.0 0.0 0 0 ? S< Aug14 0:00 [vmstat]
root 689 0.0 0.0 0 0 ? S Aug14 0:00 [fsnotify_mark]
root 708 0.0 0.0 0 0 ? S< Aug14 0:00 [kthrotld]
root 748 0.0 0.0 0 0 ? S< Aug14 0:00 [bioset]
root 752 0.0 0.0 0 0 ? S< Aug14 0:00 [deferwq]
root 1374 0.0 0.0 0 0 ? S< Aug14 0:00 [ata_sff]
root 1387 0.0 0.0 0 0 ? S Aug14 0:00 [scsi_eh_0]
root 1396 0.0 0.0 0 0 ? S< Aug14 0:00 [scsi_tmf_0]
root 1399 0.0 0.0 0 0 ? S Aug14 0:00 [scsi_eh_1]
root 1412 0.0 0.0 0 0 ? S< Aug14 0:00 [scsi_tmf_1]
root 1470 0.0 0.0 0 0 ? S Aug14 0:34 [jbd2/xvda1-8]
root 1471 0.0 0.0 0 0 ? S< Aug14 0:00 [ext4-rsv-conver]
root 1499 0.0 0.0 0 0 ? S< Aug14 0:00 [kworker/0:1H]
root 1517 0.0 0.0 11424 28 ? Ss Aug14 0:00 /sbin/udevd -d
root 1590 0.0 0.0 0 0 ? S< Aug14 0:00 [kpsmoused]
root 1641 0.0 0.0 11312 32 ? S Aug14 0:00 /sbin/udevd -d
root 1827 0.0 0.0 0 0 ? S Aug14 0:00 [kauditd]
root 1882 0.0 0.0 0 0 ? S< Aug14 0:00 [ipv6_addrconf]
root 2035 0.0 0.0 9360 628 ? Ss Aug14 0:00 /sbin/dhclient -q -lf /var/lib/dhclient/dhclient-eth0.leases -pf /var/run/dhclient-eth0.pid eth0
root 2087 0.0 0.0 46536 36 ? S<sl Aug14 0:00 auditd
root 2108 0.0 0.0 247464 0 ? Sl Aug14 0:00 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
dbus 2137 0.0 0.0 21792 4 ? Ss Aug14 0:00 dbus-daemon --system
root 2266 0.0 0.0 77840 780 ? Ss Aug14 0:00 /usr/sbin/sshd
root 2327 0.0 0.0 115344 28 ? S Aug14 0:00 /bin/sh /usr/libexec/mysql55/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid --basedir=/usr --user=mysql
mysql 2536 0.6 1.3 886436 13936 ? Sl Aug14 83:28 /usr/libexec/mysql55/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
root 2585 0.0 0.0 88992 552 ? Ss Aug14 0:18 sendmail: accepting connections
smmsp 2594 0.0 0.0 80440 400 ? Ss Aug14 0:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
root 2605 0.0 0.1 431276 1284 ? Ss Aug14 0:23 /usr/sbin/httpd
root 2616 0.0 0.0 119516 768 ? Ss Aug14 0:03 crond
root 2630 0.0 0.0 17068 88 ? Ss Aug14 0:00 /usr/sbin/atd
root 2653 0.0 0.0 4312 32 tty1 Ss+ Aug14 0:00 /sbin/mingetty /dev/tty1
root 2655 0.0 0.0 6460 32 ttyS0 Ss+ Aug14 0:00 /sbin/agetty ttyS0 9600 vt100-nav
root 2657 0.0 0.0 4312 32 tty2 Ss+ Aug14 0:00 /sbin/mingetty /dev/tty2
root 2660 0.0 0.0 4312 32 tty3 Ss+ Aug14 0:00 /sbin/mingetty /dev/tty3
root 2664 0.0 0.0 4312 32 tty4 Ss+ Aug14 0:00 /sbin/mingetty /dev/tty4
root 2666 0.0 0.0 4312 32 tty5 Ss+ Aug14 0:00 /sbin/mingetty /dev/tty5
root 2668 0.0 0.0 10876 32 ? S Aug14 0:00 /sbin/udevd -d
root 2669 0.0 0.0 4312 32 tty6 Ss+ Aug14 0:00 /sbin/mingetty /dev/tty6
root 2797 0.0 0.0 0 0 ? S Aug14 0:12 [kworker/u30:2]
ntp 2821 0.0 0.1 29288 1808 ? Ss Aug14 0:01 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
apache 12405 0.1 4.2 471264 42920 ? S Aug22 1:08 /usr/sbin/httpd
apache 12412 0.1 3.6 466372 37440 ? S Aug22 1:07 /usr/sbin/httpd
apache 12413 0.1 4.1 487140 42788 ? S Aug22 1:09 /usr/sbin/httpd
apache 12416 0.1 3.8 466764 39020 ? S Aug22 1:12 /usr/sbin/httpd
apache 12417 0.1 4.3 474916 44180 ? S Aug22 1:09 /usr/sbin/httpd
apache 12418 0.1 3.9 466596 40108 ? S Aug22 1:07 /usr/sbin/httpd
apache 12419 0.1 4.1 483704 42392 ? S Aug22 1:08 /usr/sbin/httpd
root 12960 0.0 0.0 0 0 ? S Aug22 0:00 [kworker/u30:0]
apache 14977 0.7 4.2 485100 43372 ? S Aug21 24:02 /usr/sbin/httpd
apache 14978 0.7 4.1 489860 42536 ? S Aug21 23:32 /usr/sbin/httpd
apache 14979 0.7 4.3 484092 44816 ? S Aug21 23:51 /usr/sbin/httpd
apache 14980 0.7 4.1 491120 42312 ? S Aug21 24:01 /usr/sbin/httpd
apache 14981 0.7 4.1 491984 42732 ? S Aug21 23:34 /usr/sbin/httpd
apache 14982 0.7 4.2 491680 43680 ? S Aug21 23:43 /usr/sbin/httpd
apache 14983 0.7 4.1 491768 42404 ? S Aug21 23:28 /usr/sbin/httpd
apache 14984 0.7 4.0 490436 41508 ? S Aug21 23:42 /usr/sbin/httpd
apache 15030 0.7 4.3 486920 44224 ? S Aug21 23:39 /usr/sbin/httpd
apache 15040 0.6 4.7 486704 48124 ? S Aug21 23:11 /usr/sbin/httpd
apache 15343 0.6 4.0 484224 41632 ? S Aug21 22:56 /usr/sbin/httpd
apache 16238 0.6 4.1 486556 41812 ? S Aug21 22:34 /usr/sbin/httpd
root 17169 0.0 0.6 115740 6536 ? Ss 10:58 0:00 sshd: ec2-user [priv]
ec2-user 17171 0.0 0.5 115740 5140 ? S 10:58 0:00 sshd: ec2-user@pts/0
ec2-user 17172 0.0 0.3 115476 3320 pts/0 Ss 10:58 0:00 -bash
ec2-user 17227 4.0 0.2 117204 2340 pts/0 R+ 10:58 0:00 ps aux
ec2-user 21561 0.0 0.0 113084 0 ? S Aug21 0:00 /bin/sh /home/ec2-user/bin/startJayServer.sh
ec2-user 21562 49.2 7.2 2283928 73668 ? Sl Aug21 1393:33 java -server com.jayavon.game.server.MyServer -PROD
apache 24084 0.5 4.4 486596 45336 ? S Aug21 14:53 /usr/sbin/httpd
KisnardOnline
Posted 2016-08-23T03:47:53.460
Reputation: 127
can you [edit] in the relevant output of
ps aux && free? – Matt Clark – 2016-08-23T04:41:32.610Note also that MySQL Server is often the victim, not the cause, of memory exhaustion issues.
– Michael - sqlbot – 2016-08-23T10:42:21.810@MattClark I have added in the outputs that you requested. Sorry it took so long, I was sleeping. Really appreciate your help! – KisnardOnline – 2016-08-23T15:05:04.200
From the output of
ps aux, it would appear that all of your RAM is being eaten by httpd. In the printout, I can see ~19 httpd process consuming ~40MiB each =~ 750+MiB. – Matt Clark – 2016-08-23T16:48:59.917You may want to lookup tuning guides for httpd / apache(2). – Matt Clark – 2016-08-23T16:49:39.120
Thanks Matt - I will look into that! If you turn this into an answer I will mark it accepted :) – KisnardOnline – 2016-08-23T17:58:38.180