Apache + Tomcat giving error 500 for non-browser agents (Struts2)

Question

I've got an Apache WebServer + Tomcat 7 serving Struts2 web pages. It works without problems when I use any web browser, but when I try to access a web page using W3C validator or Facebook Lint, the server returns the error 500.

[28/Oct/2011:21:45:58 +0100] "GET /agenda7/event?id=124 HTTP/1.1" 200 9898 
"-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) 
AppleWebKit/534.51.22 (KHTML, like Gecko) Version/5.1.1 Safari/534.51.22"

[28/Oct/2011:21:47:16 +0100] "GET /agenda7/event?id=124 HTTP/1.0" 500 6287 "-" 
"facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"

[28/Oct/2011:21:47:42 +0100] "GET /agenda7/event?id=124 HTTP/1.1" 500 6287 "-" 
"W3C_Validator/1.2"

Somehow, it fells like those services cannot understand how to make the get request to a web page without extension, but, if that were the case, shouldn't the error be 404?

Any suggestions?

EDIT:

Just tested this with an alternative mapping, and the problem persists:

[28/Oct/2011:22:54:53 +0100] "GET /agenda7/event_124 HTTP/1.0" 500 6364 "-" 
"facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"

But it works with the index page, for example-:

[28/Oct/2011:22:59:01 +0100] "GET /agenda7/ HTTP/1.0" 200 20666 "-"
"facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"

Answer 1

A 500 status code means that the code running within Tomcat errored; you should find a stack trace in the logs.

Keep in mind that the requests from those non-browsers are going to be different in subtle ways; the Facebook hit is HTTP/1.0, for example.

It's likely that your code is looking for something in request headers or cookies that is there with the browser request and not there in the bot requests.

Answer 2

I bet you the website is looking for headers that the bots aren't setting.