0

I've got an Apache server hosting several json APIs.

This server is running Ubuntu 16.04.4 LTS with Apache and Mysql.

Response times for all API are very good except for one! It takes about 5s to get a json response from the particular API whereas it takes milliseconds for the other APIs.

I checked multiple things without success :

1) the SQL queries ran almost instantly (with and without cache - I tried both)

2) our developper added a custom header to find out how long the API takes to generate the json response : 2-3 ms

3) calling the API from a remote computer or locally on the server with wget gives the same result : it takes 5s to get the reply. According to the browser developper mode, the 5s are spent "waiting". Google Chrome says "Waiting for the first byte". Then data transfer seems fast according to the dev mode (a few ms)

4) apache doesn’t seem to be overwhelmed by https requests, see screenshot displaying server-status page Server status

5) access to the https API is done through multiple firewalls but no reverse proxy is present

As part of the debugging process, I tried to improve the server performances :

  • I moved the vm to a our least busy VMware host

  • I moved this vm to an SSD based datastore

Do you have any idea why Apache is so slow to reply ? Do you know any way to troubleshoot this issue ?

Thank you for your help

Thomas

ThomasP
  • 1
  • 3

1 Answers1

0

Days ago i found the following thread : Apache has a long lag before responding This problem looked like mine but it sounded related to IPv6. As IPv6 is disabled on my servers, I thought the solution wasn’t suitable to solve my issue.

In a last desperate attempt to solve this issue, I tried the described solution anyway : I just added “options single-request-reopen” to /etc/resolv.conf and then the 5s delay disappeared.

At this moment, I didn’t really understand why this setting was solving my performance issue as IPv6 is disabled on my servers.

Moreover hostname lookups is disabled in Apache.

After a closer investigation ….

1) This new Json API uses Google Firebase for authentication, which means that DNS resolution is required to access Google services. I forgot this in my former investigation… I thought that DNS couldn't be a problem here. I was wrong.

2) IPv6 queries on a IPv4-only server

Basically a Linux server relies on getaddrinfo for dns resolution. According the man page (gai.conf which is getaddrinfo configuration file) :

“A call to getaddrinfo(3) might return multiple answers. According to RFC 3484 these answers must be sorted so that the answer with the highest success rate is first in the list. The RFC provides an algorithm for the sorting.”

According to the RFC 3484 :

“another effect of the default policy table is to prefer communication using IPv6 addresses to communication using IPv4 addresses,»

Back to “single-request-reopen” :

“The resolver uses the same socket for the A and AAAA requests. Some hardware mistakenly sends back only one reply. When that happens the client system will sit and wait for the second reply. Turning this option on changes this behavior so that if two requests from the same port are not handled correctly it will close the socket and open a new one before sending the second request.”

I think that my server was waiting on the second reply … which took 5 seconds. “single-request-reopen” allowed me a bypass this issue.

3) Prioritize IPv4 dns queries :

I also changed getaddinfo behavior as well to ensure the IPv4 communications are preferred, see /etc/gai.conf

# For sites which prefer IPv4 connections change the last line to

#

precedence ::ffff:0:0/96  100

Thomas

ThomasP
  • 1
  • 3