0

We're load testing a MIG (with 2 instances) hosted behind the HTTPs load balancer using JMeter.

Observation 1: We randomly receive 404 error and 503 error, for 404 we see an entry get created within load balancer monitoring NO_BACKEND_SELECTED (other than our actual MIG backend). Further, for 503 we see an entry get created within load balancer monitoring FRONTEND_5XX.

Based on GCP:

NO_BACKEND_SELECTED - An error or other interruption occurred before a backend could be selected. FRONTEND_5XX - An internal error occurred before the GFE could select a backend. The GFE returned 5XX to the client.

The above statement doesn't assist with respect to troubleshooting or getting the same resolved or isolation of the cause for the issue. we didn't find anything w.r.t. these error messages within GCP docs or other articles.

Observation 2: We randomly receive random SSL exceptions; Remote host terminated connection, read handshake, socket closed & upstream connect @ JMeter's end.

Steps taken

  1. Changing Keep Alive on the backend servers to 620 sec (GFE has Keep Alive of 600 secs)
  2. Created custom SSL policy (minimum tls set to 1.1)
  3. Increased the backend timeout from default 30 to 65 seconds

So, we are looking @ what are we missing or what else can we fine-tune/modify for testing purposes in order to get the above mentioned issues resolved.

Thank you. Gaurav_N17

Luis Daniel Albarran
  • 51
  • 1
Gaurav_N17
  • 1
  • 1

2 Answers2

0

It sounds like sometimes might have no healthy instances on the backend.

I suggest you to Monitor autoscaling charts and logs and take a look at Commonly returned status message.

These steps are supposed to give you a clue about what's going on on backend and load balancer

surfingonthenet
  • 695
  • 2
  • 6
  • we've already analyzed that during load testing and there's no issue with instances running at the backend. Any idea w.r.t. NO_BACKEND_SELECTED and FRONTEND_5XX error messages. – Gaurav_N17 Feb 18 '22 at 06:16
  • @Gaurav_N17, I'm wondering if backend instances might have gotten overloaded at load testing time, and got unable to answer request due to high load (CPU, I/O, processing, max HTTP open connections, etc). It's not a application issue, but a max resources limitation – surfingonthenet Feb 18 '22 at 21:24
  • yes looks like the same, we observed high load within the database servers during the load test. We'll be increasing the resources and then carry out another set of load testing. – Gaurav_N17 Feb 21 '22 at 06:20
0

after load testing, we concluded that it was at the high resource utilization at the database end which was causing the issue. we vertically scaled the DB resources. thank you.

Gaurav_N17
  • 1
  • 1