16

So, if I do some benchmarking with apache benchmark (ab), and I use large numbers of requests. Then sometimes in the middle of a test I get this error.

I don't even know what it means. So how can I fix it? Or is it just something that will happen if the server gets too many hits anyway? The problem is, if I run 10,000 hits, it'll all run perfectly. If I run it again, it'll get to 4000 and get the error:

apr_socket_recv: Connection reset by peer (104)

A little about my setup: I have nginx taking static requests and processing dynamic ones to apache. The file in question is served from cache by nginx, so I guess it's probably got to do with how nginx is handling the requests?

Ideas?

Matthew
  • 1,769
  • 4
  • 21
  • 32

5 Answers5

8

The error means that the other end (webserver) suddenly disconnected in the middle of the session. have a look at the apache or nginx error logs to see if there is anything suspicious there.

Aleksandar Ivanisevic
  • 3,327
  • 19
  • 24
4

It means that server is heavly loaded with the request i.e, all the threads are busy serving the request. Solution : either increase the maxThread attribute count for connector in server.xml file or increase acceptCount attribute value.

acceptcount : The maximum queue length for incoming connection requests when all possible request processing threads are in use. Any requests received when the queue is full will be refused.

Kushal Bafna
  • 41
  • 1
1

Besides the answers here, I have read a lot of other ones:

None of them helped.

I thought about switching to wrk after seeing similar struggles.

Finding the problem

The problem seems to be related to the amount of ephermal ports. I tried to set it from 50000 to 25000 as this is the port range. Still no luck. Then I got the impression it is related to TIME_WAIT and this blog post. I think I could confirm that:

$ netstat -nat | awk '{print $6}' | sort | uniq -c | sort -n

    1 CLOSE_WAIT
    1 established)
    1 Foreign
    4 LISTEN
    8 SYN_SENT
   62 SYN_RECV
  351 ESTABLISHED
13916 TIME_WAIT

What I tried

I didn't fix it so far :-/

According to sudo sysctl -a | grep net.ipv4.tcp, I have:

net.ipv4.tcp_tw_reuse = 0    # No luck setting only that to 1
net.ipv4.tcp_max_tw_buckets = 32768
net.ipv4.tcp_fin_timeout = 60  # Setting it to 5 didn't help either
Martin Thoma
  • 247
  • 4
  • 13
0

I had same problem and my server version was:

Server Version: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 PHP/5.6.5 mod_perl/2.0.9dev Perl/v5.16.3

i removed unneccessary modules and problem is gone:

Server Version: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips

So one of mod_fcgid, mod_php or mod_perl is causing problem. You can try to disable those if you are not using.

(Side note; If you are using opcache,disable fast_shutdown too. It was causing problem too: opcache.fast_shutdown=0 )

Ünsal Korkmaz
  • 63
  • 3
  • 12
0

This issue is caused by the system. if give a high concurrency request to the system. OS kernel will trigger SYN flood protection. So the system will reset the link. you can modify the OS config in the file.

#vi /etc/sysctl.conf
net.ipv4.tcp_syncookies = 0 # set value is 0
#sysctl -p # read config from the config file.

you can try it.

usually the attribute net.ipv4.tcp_syncookies was used to protect OS to avoid the huge request attack. But if you want to use this OS to do some Load test or performance test, you should close this feature.

Jenny D
  • 27,358
  • 21
  • 74
  • 110
moneyfly
  • 11