0

This server I am benchmarking gives no obvious signs of getting overloaded at a first glance.

top output:

top - 22:48:25 up 5 days,  3:41,  3 users,  load average: 3.19, 3.34, 3.25
Tasks: 146 total,   1 running, 145 sleeping,   0 stopped,   0 zombie
Cpu(s): 29.4%us, 14.1%sy,  0.0%ni, 54.0%id,  0.0%wa,  0.0%hi,  2.5%si,  0.0%st
Mem:  24158176k total, 15366776k used,  8791400k free,   385372k buffers
Swap:  1952764k total,        0k used,  1952764k free,   994752k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                                              
22913 root      20   0 21.5g  10g  10m S  737 47.1 446:13.10 java

/var/log/syslog is just as it has always been. Server being benchmarked has some logs that give no clues to the reason for which requests are becoming increasingly more slowly processed (from standard of 10 - 50 ms into 50 seconds and over). This is a java server, no GCs taking place whatsoever.

Just what else is there to keep an eye onto in such scenario ?

kellogs
  • 69
  • 8
  • 1
    Does the server talk to anything else, e.g. a database? How many connections do you have simultaneously? How are the disks doing? – Jenny D Dec 17 '13 at 15:10
  • DNS resolution is another thing to look at. Some applications (notably apache) can attempt to do a dns lookup on connecting hosts. It might not your host at all. – GeoSword Dec 17 '13 at 15:17
  • Looks overloaded to me. Try running `top` and pressing `1` and `H` and displaying that. – Matthew Ife Dec 17 '13 at 15:19
  • @JennyD Yes it talks to an even less loaded (according to top command) DB server through about 12 connections and toa virtually no-load dummy HTTP server across some 10k connections. disks ? Hmm, I have no business with them and swap looks fine. Where would I look for anomalies ? – kellogs Dec 17 '13 at 15:19
  • @GeoSword No DNS involved here – kellogs Dec 17 '13 at 15:20
  • Does the java server try to open more than 12 connections/is there any connection limit in the DB server? As for disks, look at e.g. `iostat` – Jenny D Dec 17 '13 at 15:21
  • @MIfe I have failed to point out the underlaying machine is 16 core. – kellogs Dec 17 '13 at 15:22
  • @JennyD Whenever there are DB issues this java server logs them thoroughly. Not the case in this test. Besides, DB server's machine is mostly relaxed (about 3.0 load maximum, 16 cores). Also, "show processlist" command does not hint towards any serious usage. About iostat, is that not a realtime tool ? The overload conditions are now long gone. So it would make more sense to inspect some I/O logs if there are any. Besides, not sure how to use / interpret iostat. – kellogs Dec 17 '13 at 15:26
  • Please provide also the output of `cat /proc/loadavg` – Matthew Ife Dec 17 '13 at 15:27
  • @MIfe HEre it is: root@XMPP:/opt/tigase# cat /proc/loadavg 2.57 2.49 3.10 5/1322 28667 root@XMPP:/opt/tigase# cat /proc/loadavg 2.41 2.46 3.08 3/1322 28674 . It appears these are realtime stats too just like top outputs. At the moment irrelevant as server is not overloaded any longer. Is there anything wrong with top first line of output ? – kellogs Dec 17 '13 at 15:28
  • Sys is very high, what is the output of `perf top` – Matthew Ife Dec 17 '13 at 15:34
  • @MIfe will take some time to install that package. Machine is having problems with apt-get. – kellogs Dec 17 '13 at 15:41
  • tried fiddling with iptables rules but still cant download from ubuntu repository IP address.I can however SSH into that server from here. So will try fetching some linux-tools package and drop it over there. Not likely to work out though :( – kellogs Dec 17 '13 at 16:03

0 Answers0