I'm trying to work out why my server has such a high CPU consumption coming from Apache 2.4.I'm on Ubuntu 16.
Here is what I currently see in "top":
root@mail:~# top
top - 12:54:47 up 42 min, 2 users, load average: 1.96, 1.94, 1.97
Tasks: 178 total, 3 running, 175 sleeping, 0 stopped, 0 zombie
%Cpu(s): 43.5 us, 56.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 2044912 total, 214760 free, 1316416 used, 513736 buff/cache
KiB Swap: 262140 total, 255940 free, 6200 used. 638968 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6541 www-data 20 0 721368 112640 8528 R 49.8 5.5 1:16.68 /usr/sbin/apach
6542 www-data 20 0 522556 103532 3504 R 49.8 5.1 1:16.60 /usr/sbin/apach
3532 mysql 20 0 1112464 151152 17516 S 0.3 7.4 0:01.45 mysqld
3674 jetty 20 0 2329684 213996 18208 S 0.3 10.5 0:15.11 java
4526 root 20 0 95404 6776 5836 S 0.3 0.3 0:00.16 sshd
5866 root 20 0 41816 3888 3172 R 0.3 0.2 0:01.02 top
After looking around online, I found about the strace -p PID
feature. I run that on one of the offending processes, and I hundreds of thousands of this:
read(2, 0x7ffcae1de7d3, 1) = -1 EBADF (Bad file descriptor)
read(2, 0x7ffcae1de7d3, 1) = -1 EBADF (Bad file descriptor)
read(2, 0x7ffcae1de7d3, 1) = -1 EBADF (Bad file descriptor)
read(2, 0x7ffcae1de7d3, 1) = -1 EBADF (Bad file descriptor)
read(2, 0x7ffcae1de7d3, 1) = -1 EBADF (Bad file descriptor)
read(2, 0x7ffcae1de7d3, 1) = -1 EBADF (Bad file descriptor)
read(2, 0x7ffcae1de7d3, 1) = -1 EBADF (Bad file descriptor)
read(2, 0x7ffcae1de7d3, 1) = -1 EBADF (Bad file descriptor)
read(2, 0x7ffcae1de7d3, 1) = -1 EBADF (Bad file descriptor)
read(2, 0x7ffcae1de7d3, 1) = -1 EBADF (Bad file descriptor)
I tried to get back to the very start of the process (to see what is happening), but I can't respond quick enough!
It's got 2Gb RAM and 1 CPU, with an SSD drive.
As suggested, I tested the same processes with gdb, and got this error:
../sysdeps/unix/syscall-template.S: No such file or directory.
I came across this post:
They suggest doing:
sudo apt-get install eglibc-source
I couldn't find that package, but did find glibc-source
, so have installed that. Unfortunately I still get the same error (even after a restart);
84 ../sysdeps/unix/syscall-template.S: No such file or directory.
Doing some more digging, I can see:
root@mail:~# gdb --pid=7708
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 7708
[New LWP 7712]
[New LWP 7713]
[New LWP 7714]
[New LWP 7715]
[New LWP 7716]
[New LWP 7717]
[New LWP 7718]
[New LWP 7719]
[New LWP 7720]
[New LWP 7721]
[New LWP 7722]
[New LWP 7723]
[New LWP 7724]
[New LWP 7725]
[New LWP 7726]
[New LWP 7727]
[New LWP 7728]
[New LWP 7729]
[New LWP 7730]
[New LWP 7731]
[New LWP 7732]
[New LWP 7733]
[New LWP 7734]
[New LWP 7735]
[New LWP 7736]
[New LWP 7737]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fcc946f051d in read () at ../sysdeps/unix/syscall-template.S:84
84 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) n
86 in ../sysdeps/unix/syscall-template.S
(gdb) n
0x000055b4134ed9b7 in ap_mpm_podx_check ()
(gdb) n
Single stepping until exit from function ap_mpm_podx_check,
which has no line number information.
0x00007fcc9163935b in ?? () from /usr/lib/apache2/modules/mod_mpm_event.so
(gdb) n
Cannot find bounds of current function
(gdb)
I also have glibc-source installed, so I'm confused as to why it would even have this error:
root@mail:~# apt-get install glibc-source
Reading package lists... Done
Building dependency tree
Reading state information... Done
glibc-source is already the newest version (2.23-0ubuntu7).
0 upgraded, 0 newly installed, 0 to remove and 7 not upgraded.
If I do a locate syscall-template.S
I get no results (even after running updatedb
), so I'm guessing it IS missing, but I'm unsure why)
I'm still not sure what that mean's though :/