5

I've read through other threads on performance differences between RHEL 6 and RHEL 5, but none seem a tight match to mine.

My issue manifests itself in slightly slower average response time (20ms) per request. I have about 10/10 servers of the same hardware spec with CentOS 6.1 and CentOS 5.6. The issue is consistent across the group.

I am running Ruby on Rails with Passenger.

  • Apache config is identical (checked out from the same SVN repo)
  • Ruby and Passenger are identical builds.
  • Application is identical and being served traffic round robin.
  • mod_worker

An interesting clue from server-status: The CentOS 6.1 servers have a steady 20-40 threads in the "Reading Request" state while the CentOS 5.6 servers have around 1. I'm graphing this so I can see it trend over time.

I also have a bunch of much newer machines that are significantly faster and are running CentOS 6.1. They dust all the older machines in response time, but I can see they also have a steady 20-40 threads in the "Reading Request" state. This makes me believe I can get their response time down, if I can figure out what is holding up these requests.

My gut is telling me that I need to tune some network setting in sysctl, but I haven't figured it out yet.

Cristian Ciupitu
  • 6,226
  • 2
  • 41
  • 55
daniel
  • 51
  • 1
  • ooh. interesting one. There are bonkers loads of metrics and settings that you could pull here to find out. – Tom May 09 '12 at 00:06
  • How many requests per second are you serving avg? whats the current cache and buffers? – Tom May 09 '12 at 00:14
  • each individual server gets to around 40/sec – daniel May 09 '12 at 00:35
  • there is a good amount of free memory, if that's what you are asking CentOS6: MemTotal: 8061920 kB MemFree: 1898468 kB Buffers: 92748 kB Cached: 2422552 kB CentOS 5.6: MemTotal: 8174968 kB MemFree: 1107736 kB Buffers: 147204 kB Cached: 4750620 kB – daniel May 09 '12 at 00:37
  • I would probably dive into the guts of apache and see what syscalls these server-status statuses correspond to, at the moment I am only commenting because I can't spare the time, but if you are still at a loss later in the week ill take a look when i am back in the dc – Tom May 09 '12 at 00:59
  • Cool Tom, I just did a side-by-side sysctl -a and didn't find much. Apache is my next step. thanks! – daniel May 09 '12 at 01:11
  • 1
    Apache versions? – Mike Fiedler May 09 '12 at 02:13
  • CentOS6 uses 2.2.15 and CentOS5 uses httpd-2.2.3 – daniel May 09 '12 at 02:21
  • Any news on this? – Cristian Ciupitu Jul 05 '14 at 10:08

2 Answers2

0

One FAQ in this area is that memory mgmt in RHEL6 is different and mulit-threaded apps can suffer. Try running apache under MALLOC_ARENA_MAX=1 and see if it makes a difference.

covener
  • 1,665
  • 9
  • 15
0

Wild shot #1: Any chance you're logging hostnames instead of IP addresses? If yes is that on both? If yes is one of them running nscd?

Wild shot #2: Is LDAP involved in any way there? I.e. is the apache or any other component of the request running as an ldap user?

V13
  • 231
  • 1
  • 5