1

I'm trying to work out why my server has such a high CPU consumption coming from Apache 2.4.I'm on Ubuntu 16.

Here is what I currently see in "top":

root@mail:~# top
top - 12:54:47 up 42 min,  2 users,  load average: 1.96, 1.94, 1.97
Tasks: 178 total,   3 running, 175 sleeping,   0 stopped,   0 zombie
%Cpu(s): 43.5 us, 56.5 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  2044912 total,   214760 free,  1316416 used,   513736 buff/cache
KiB Swap:   262140 total,   255940 free,     6200 used.   638968 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 6541 www-data  20   0  721368 112640   8528 R 49.8  5.5   1:16.68 /usr/sbin/apach
 6542 www-data  20   0  522556 103532   3504 R 49.8  5.1   1:16.60 /usr/sbin/apach
 3532 mysql     20   0 1112464 151152  17516 S  0.3  7.4   0:01.45 mysqld
 3674 jetty     20   0 2329684 213996  18208 S  0.3 10.5   0:15.11 java
 4526 root      20   0   95404   6776   5836 S  0.3  0.3   0:00.16 sshd
 5866 root      20   0   41816   3888   3172 R  0.3  0.2   0:01.02 top

After looking around online, I found about the strace -p PID feature. I run that on one of the offending processes, and I hundreds of thousands of this:

read(2, 0x7ffcae1de7d3, 1)              = -1 EBADF (Bad file descriptor)
read(2, 0x7ffcae1de7d3, 1)              = -1 EBADF (Bad file descriptor)
read(2, 0x7ffcae1de7d3, 1)              = -1 EBADF (Bad file descriptor)
read(2, 0x7ffcae1de7d3, 1)              = -1 EBADF (Bad file descriptor)
read(2, 0x7ffcae1de7d3, 1)              = -1 EBADF (Bad file descriptor)
read(2, 0x7ffcae1de7d3, 1)              = -1 EBADF (Bad file descriptor)
read(2, 0x7ffcae1de7d3, 1)              = -1 EBADF (Bad file descriptor)
read(2, 0x7ffcae1de7d3, 1)              = -1 EBADF (Bad file descriptor)
read(2, 0x7ffcae1de7d3, 1)              = -1 EBADF (Bad file descriptor)
read(2, 0x7ffcae1de7d3, 1)              = -1 EBADF (Bad file descriptor)

I tried to get back to the very start of the process (to see what is happening), but I can't respond quick enough!

It's got 2Gb RAM and 1 CPU, with an SSD drive.

As suggested, I tested the same processes with gdb, and got this error:

../sysdeps/unix/syscall-template.S: No such file or directory.

I came across this post:

http://www.squaregoldfish.co.uk/2013/01/06/debugging-in-ubuntu-12-10-missing-file-syscall-template-s/

They suggest doing:

sudo apt-get install eglibc-source

I couldn't find that package, but did find glibc-source, so have installed that. Unfortunately I still get the same error (even after a restart);

84      ../sysdeps/unix/syscall-template.S: No such file or directory.

Doing some more digging, I can see:

root@mail:~# gdb --pid=7708
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 7708
[New LWP 7712]
[New LWP 7713]
[New LWP 7714]
[New LWP 7715]
[New LWP 7716]
[New LWP 7717]
[New LWP 7718]
[New LWP 7719]
[New LWP 7720]
[New LWP 7721]
[New LWP 7722]
[New LWP 7723]
[New LWP 7724]
[New LWP 7725]
[New LWP 7726]
[New LWP 7727]
[New LWP 7728]
[New LWP 7729]
[New LWP 7730]
[New LWP 7731]
[New LWP 7732]
[New LWP 7733]
[New LWP 7734]
[New LWP 7735]
[New LWP 7736]
[New LWP 7737]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fcc946f051d in read () at ../sysdeps/unix/syscall-template.S:84
84      ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) n
86      in ../sysdeps/unix/syscall-template.S
(gdb) n
0x000055b4134ed9b7 in ap_mpm_podx_check ()
(gdb) n
Single stepping until exit from function ap_mpm_podx_check,
which has no line number information.
0x00007fcc9163935b in ?? () from /usr/lib/apache2/modules/mod_mpm_event.so
(gdb) n
Cannot find bounds of current function
(gdb)

I also have glibc-source installed, so I'm confused as to why it would even have this error:

root@mail:~# apt-get install glibc-source
Reading package lists... Done
Building dependency tree
Reading state information... Done
glibc-source is already the newest version (2.23-0ubuntu7).
0 upgraded, 0 newly installed, 0 to remove and 7 not upgraded.

If I do a locate syscall-template.S I get no results (even after running updatedb), so I'm guessing it IS missing, but I'm unsure why)

I'm still not sure what that mean's though :/

Andrew Newby
  • 1,041
  • 1
  • 22
  • 48
  • You dont read from stderr normally (file descriptor 2 is stderr). Fetch the apache debug package and run it through `gdb --pid=` then use the `bt` command to print a backtrace. – Matthew Ife Mar 31 '17 at 13:43
  • @MatthewIfe - not sure what you mean? – Andrew Newby Mar 31 '17 at 13:44
  • @MatthewIfe - thanks, that is getting me a bit closer. I have updated my opening question now. Basically, I see this error: `84 ../sysdeps/unix/syscall-template.S: No such file or directory.` – Andrew Newby Mar 31 '17 at 13:52
  • https://bz.apache.org/bugzilla/show_bug.cgi?id=57746 – Matthew Ife Mar 31 '17 at 15:36
  • @MatthewIfe - thanks. I'm not too sure how to make that change (background). I'm amazed that is a bug still 2 years on as well :/ – Andrew Newby Mar 31 '17 at 15:40
  • @MatthewIfe I'm coming back to this today with a fresh mind. I'm trying to work out how you invoke Apache with the `-DFOREGROUND` flag set. I've tried: `apache2ctl start -DFOREGROUND` and `apache2ctl -DFOREGROUND start`, but both fail (just comes up with the help section as if you had run `apache2ctl -h` – Andrew Newby Apr 01 '17 at 06:29
  • I'm pretty sure that this is a bug due to multi-threading not guarding a key variable somewhere. You should never be reading from file descriptor 2 (thats standard error normally). You'd need to learn a lot about how apache is designed for mpm_event in order to fix that. – Matthew Ife Apr 01 '17 at 09:58
  • @MatthewIfe - thanks, it's well over what I know how to do :( Is there a way to disable the multi-threading? I tried posting on the bug report, but they want you to actually sign up to a whole usergroup just to post on it! – Andrew Newby Apr 01 '17 at 11:54
  • Try using mpm_worker or mpm_prefork (not recommended). Event by and large is way more scalable though. If you still are having no luck try nginx instead which is also a (better) event based web server. – Matthew Ife Apr 01 '17 at 16:49
  • @MatthewIfe - thanks. My server tech friend did this for me yesterday, and it works like a charm now :) Why would you say its not recommended? Also, I'm already using nginx (to server static), but the reason I'm also having to use Apache2, is that mod_perl isn't a feature in nginx for some reason, and that leads to less than desirable load times for my dynamic pages. Please add the "prefork" suggestion as an answer, and I will accept it :) – Andrew Newby Apr 03 '17 at 06:10

0 Answers0