socket timeout exception on ubuntu

Question

We have solr router and shards. This appears in the jetty log of the router when doing faceted search across all the shards.

org.apache.commons.httpclient.HttpMethodDirector executeWithRetry 
INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error 
org.apache.commons.httpclient.HttpMethodDirector executeWithRetry 
INFO: Retrying request

followed by exception about Internal Server Error. We run eighty shards distributed across several servers. Router runs on its own node.

UPD: In the HttpCommComponent class of SOLR which sumbits requests to shards using HttpClient class I have noticed the following set on the MultiThreadedHttpConnectionManager: mgr.getParams().setDefaultMaxConnectionsPerHost(20);

it seems to me like too little setting for 80+ shards.

@tristan http://pastebin.com/FTVH1WF3 that is solr router log — D_K, May 02 '13 at 13:58
@tristan http://pastebin.com/exwCx3LX that is the shard log, which port is mentioned in the solr router log. — D_K, May 02 '13 at 13:59

score 1 · Answer 1 · answered May 02 '13 at 13:36

1

The message is saying that your sever received a tcp-reset and therefore cannot continue.

I assume he was acting as a HTTP client? Is the target URL correct (ie not a closed port).

Using tcpdump or similar you should be able to figure out what went wrong.

answered May 02 '13 at 13:36

Zabuzzman

733
10
25

do you suggest running tcpdump on the node with the router or on the shards? – D_K May 02 '13 at 15:40
ok. So ran tcpdump on the shards servers and listened to the router node. I get occasionally these: shard_node.9199 > router_node.45503: Flags [P.], cksum 0xab1a (incorrect -> 0x3561), seq 2896:4239, ack 1, win 634, options [nop,nop,TS val 431980810 ecr 924213364], length 1343 What does this tell? – D_K May 02 '13 at 15:48
ok. tcpdump does not show anything anomalous. Belief is that it is application (solr) level issue. – D_K May 03 '13 at 12:32
Well you should see an [R] packet no? Who's sending it when? – Zabuzzman May 03 '13 at 20:51
yes, many packet details were visible, including their hex and decoded representation. Nothing suspicious on that level. But as said, the POST packet size is limited by tomcat => see the answer. Thanks for helping! – D_K May 05 '13 at 06:57

score 0 · Accepted Answer · answered May 03 '13 at 16:39

After some more debugging I have found out, that one of the requests had a size of 4,4MB. The default maxPostSize in tomcat6 is 2MB (http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html).

Changing that to 10MB has greatly improved situation on the solr side.

socket timeout exception on ubuntu

2 Answers2