1

So.. Interesting problem for you guys, As I'm completely lost as to what to do, or where to take the next step.

Server & Application Environment.

CentOS release 5.3 (Final)

Apache 2.2.3-22

EnableSendfile off
EnableMMAP off
ErrorLog logs/error_log
LogLevel debug

PHP-5.2.6-2

error_reporting = E_ALL
display_errors = on
log_errors = on
max_execution_time=300
max_input_time=60
memory_limit=512mb

Kohana 2.3 PHP Environment.

HAProxy 1.3.15.6-2

MemCacheD 1.2.6-1

Our application is split between 3 web servers, mounting a NFS Storage server, and sticky load balancing between the 3 web servers. The application seemingly runs great, but every so often, instead of loading, the application just shows a pure white page. Not a 404 Error, or a 500 Server Error, a clean white page. And it returns instantly, so its not a execution time error.

Nothing in the Error log, or Server-Error Log, Proxy log shows standard proxied connection, Just the standard 200-Status in Access log, with 256 bytes transferred.

To me, this leads to tell me that the application itself is having a problem. A rare, unexplainable, seemingly random, problem that causes what we've now called the "White Screen of Death."

Our developers all say that since there is nothing going to our error logs, that it must be a server problem. But I say the same thing, There's nothing going to ANY of our logs (relevent to this anyway), and we're not having httpd children crash from what i can tell.

Any ideas on how i can increase my logs, or somehow prove that its not a bug in PHP, Apache, CentOS, ect? Or if it is somehow a bug, identify it?

grufftech
  • 6,620
  • 4
  • 35
  • 37
  • edited (added MemCache & that we do get a standard 200-status in access log) – grufftech Oct 28 '09 at 22:55
  • Update: Seems that the bug as "magically" disappeared... For the time being i have no clue what has fixed it, or if its even really fixed. Will update if i find anything of relevance. – grufftech Nov 03 '09 at 08:49
  • Is the problem specific to browser type or platform? So many problems I've fought are bogus as the browser was at fault with something cached. If possible, test it with something neutral like wget. – labradort Oct 28 '09 at 23:42
  • All browsers are firefox. – grufftech Jan 27 '10 at 22:29
  • Is there an error log on the Load Balancer? What is the load balancer? A commercial LB (F5)? nginx? Cheers – HTTP500 Oct 28 '09 at 22:30
  • Proxy is a standard CentOS 5.3 box running HAProxy. ( http://haproxy.1wt.eu/ ) – grufftech Oct 28 '09 at 22:56

3 Answers3

3

Telnet to the server to see what's going on

telnet server.whatever 80

GET / HTTP/1.0

Matt Simmons
  • 20,218
  • 10
  • 67
  • 114
  • Could also use Live HTTP Headers (Firefox plugin) or similar to inspect the headers being passed. – HTTP500 Oct 29 '09 at 02:49
1

I would try setting up a packet sniffer at all points in the network data path, or at least between the client and the server. You can then inspect the packets at the time the error occurs and at least isolate the problem down to one of the machines in the chain. At least then it's more clear where you should focus your investigation.

Kamil Kisiel
  • 11,946
  • 7
  • 46
  • 68
0

As it turns out (for reference) We discovered a bug (I dont know the specifics, I'm not a developer) in Kohana 2.3 with their file-caching system. By upgrading to 2.3.4, and using memcached exclusively for caching, the problem went away.

grufftech
  • 6,620
  • 4
  • 35
  • 37