5

I recently moved my site to a new server (Apache 2, PHP5, MySQL5). The site is an Invision based forum. Every few posts / topics it just hangs. The data has been written because if you stop and reload, the post / thread is there. I thought it was a write issue initially, but nope. The data is written but the page load never completes. It doesn't leave the page where the data has been input.

What's the best way to troubleshoot this issue? The only thing I have done recently is reduce my MySQL timeouts, but I can't see that being an issue as the values are still big enough and there are no mentions of timeouts in the MySQL log. For the record there is nothing in PHP's error log either.

EDIT: I checked my server-status. It all looked ok, but I have a suspicion I was hitting my ServerLimit, so I doubled that. Also enabled my Keepalives. Will keep an eye on it.

EDIT 2: It's now been a few days and this is still occuring. I have more info though;

  1. Apache is throwing seg faults, but enabling core dumps does not produce them.
  2. I have tried disabling the modules in apache but it just stops things from working.
  3. I fear it may actually be DNS related. If I watch Live Headers in Firefox, absolutely nothing happens during this 'hanging' period. After that, the responses come back fairly promptly.

UPDATE (05/04): I built the latest versions of Apache and PHP from source, no luck. I then removed those and used the remi repo to update all my packages to the latest stable. Segfaults seem to have stopped, but the hanging is continuing.

ini's are at:

  • www.skylinesaustralia.com/php.ini
  • www.skylinesaustralia.com/my.cnf
  • www.skylinesaustralia.com/httpd.conf

UPDATE - SOLVED! - The issue was having a gigantic query cache size in MySQL. It was 2GB, changing it to 64M sorted it.

brasofilo
  • 111
  • 5
Christian
  • 779
  • 1
  • 13
  • 31
  • Interesting - I wouldn't have thought a 2Gb cache would have been very expensive to look up (perhaps it doesn't have an index). What version of MySQL? What was the total blocks and free blocks when this was happenning? – symcbean Apr 14 '13 at 22:46

8 Answers8

4
  1. Apache is throwing seg faults

There's your problem. Apache isn't hanging, your browser is hanging when apache's child dies and doesn't complete the request.

LogLevel debug may give you a bit more information (though, a ton of information if this is a busy production system).

My first guess would be suhosin/mod_security or something that is compiled against the wrong php api or possibly using shared libraries that have changed slightly. I'd doublecheck those and if you can isolate the different php extensions/apache modules that are running. Are you running apcc, eaccelerator or any opcode cache? Zend? Source Guardian? ioncube? Are you using the right version of each of these? Latest updates? Zend 3.3.9, ioncube had a recent update as well and can cause segfaults depending on the version of Zend you're running.

  • Apache is no longer throwing segfaults. This stopped yesterday when I recompiled it. The hanging is continuing though and I am starting to think it might not be Apache. – Christian Apr 04 '10 at 22:09
  • Oh, I didnt answer your other queries; I stopped memcached last week and I am running the ioncube_loaders for 5.3 (was 5.1 yesterday before I upgraded php). Have updated all. – Christian Apr 04 '10 at 22:51
  • 1
    Does it happen every time you post? or post/surfing causes it? are you running suhosin? Is the server moderately busy when it happens or does it not seem to matter how much is being pushed through. Does Firefox believe it is still connected and waiting for information, i.e. transfer indicator still spinning or has it stopped? Can you roll back to php 5.2.10 (or .12)? –  Apr 04 '10 at 23:37
  • 90% of the time when posting. I found out this morning that its happening with postfixadmin, so seems to be when writing data (altho, other writes work ok as we manage sessions through MySQL and that seems ok.) No to suhosin. Prob doesnt occur when just browsing the site. So, when you click 'post' the transfer bar appears and does not fill while it hangs (1min+) and the spining load indicator does not stop spinning. If I watch 'Live HTTP Headers' I see nothing appear for 1min+. Have tried with 3 versions of PHP and same prob. – Christian Apr 05 '10 at 03:07
  • 1
    Is snort or another IDS running on the machine? You're not running mod_security that I can tell, but, it really seems like something is dropping traffic based on the length of a header request. Any firewall running on the machine or on the router/switch above? –  Apr 05 '10 at 15:15
  • No snort or IDS that I am aware of. No firewall on the machine, but will check the network above. Thanks very much btw for your help. – Christian Apr 05 '10 at 22:06
1

Bind something like strace or gdb to all of the apache worker processes, and reload pages in the browser one breaks. Then you'll get to see more info about what it's doing when it's crashing.

When you say that you've enabled coredumps, did you also enable them to be large enough to handle an Apache coredump? It's likely to be fairly large, so even setting them to unlimited-size temporarily might help.

mibus
  • 816
  • 4
  • 5
  • I have run the strace a bunch of times and there are a heap of 'permission denied' or 'no such file or directory' messages but they seem to be for obscure paths in the fs; 18616 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) I can't see anything in the trace that would look like a crash, but then I dont really know what to look for specifically. – Christian Apr 01 '10 at 23:59
1

Here is a few tips considering the high numbers of possibilities and low amount of details.

  1. Open your forum configuration file, make sure the MySQL connection host is localhost
  2. open /etc/hosts make sure you have a line like this : 127.0.0.1 localhost.localdomain localhost
  3. Shutdown any firewall you may run.
  4. Shutdown any MySQL cache engine you may have. (xcache, memcached, ...)
  5. Uninstall php-mysql package and reinstall it (or force a reinstall).

If nothing of this work, please post content of php.ini, httpd.conf and my.cnf.

Embreau
  • 1,277
  • 1
  • 9
  • 10
  • All done. I removed memcached last week as well as disabled the firewall. Reinstalled the php-mysql package this morning, no difference. Conf and hosts file are all good. Will post those files in the OP now. – Christian Apr 04 '10 at 22:04
0

If you use firefox install the Firebug plugin. Enable it by clicking the bug icon in the lower right hand corner of the browser and then set "Enable" in the Net tab. Load up a page of your forum and check for errors.

anonymouse
  • 239
  • 1
  • 2
0

There's a good chance it's a serious issue with the servers OS or hardware.

I'd try to do a verify of the os (Debian based linuxes use debsums, not sure for anything else).

Also try a run of memtestx86+ which should confirm that there's no massive memory issue, they can show up only under load.

LapTop006
  • 6,466
  • 19
  • 26
  • Thanks for the help. The machine is remote and is brand new. We have 16GB memory and most of it is used for a large MySQL db. We have no other problems at all, Apache works beautifully until data is posted through the new thread forum. – Christian Apr 01 '10 at 04:49
0

I seen similar issues related to DNS, any chance you'd post the site URL?

EDIT: Just saw you mentioned that already, run the site through http://www.intodns.com/ it could also be related to the servers hostname. Like I said if you don't mind post the domain it may be easier.

Hugh Mann
  • 386
  • 1
  • 8
  • Thanks for that link, I tried it through a few other dns checkers, but this one is the nicest by far. :) http://www.intodns.com/skylinesaustralia.com Looks like I might be able to scratch blaming DNS though. :( – Christian Apr 01 '10 at 23:44
0

Try stopping Apache, then restart it with: strace /path/to/httpd -X (only one worker will be started, and Apache won't detach from the console.) Reproduce the hang and inspect the output of strace. That should localize what part of the system is crashing.

0

Try reducing the query cache size in MySql.

Neil Knight
  • 207
  • 3
  • 10