Why does only one of the multiple address records for my AWS NLB work at a single time?

Question

I'm using an AWS network load balancer. When I issue nslookup dualstack.app.elb.us-east-2.amazonaws.com, the output is

Non-authoritative answer:
Name:   dualstack.app.elb.us-east-2.amazonaws.com
Address: 3.xxx.xxx.176
Name:   dualstack.app.elb.us-east-2.amazonaws.com
Address: 18.xxx.xxx.40

I noticed each of these IPs is in a different availability zone and that only one of the IPs is valid at a single time. Making a request to dualstack.app.elb.us-east-2.amazonaws.com/healthcheck via curl only works half of the time. However making the same request from my browser works 100% of the time because of how chrome has its own method of handling round-robin DNS (related: https://serverfault.com/a/774411, https://serverfault.com/a/852421)

Is this the intended behavior of NLB, that when multiple IPs are present, only one of them is expected to work at a time?

Answer 1

The answer above tell you why, but not how to solve it. You've said you're using an ELB, which usually refers to the classic ELB rather than ALB / NLB. With ELB you enable cross zone load balancing. It's enabled by default with an ALB, so if you're using one of those this isn't the right answer for you - please update your question and let me know if that's the case.

aws elb modify-load-balancer-attributes --load-balancer-name my-loadbalancer --load-balancer-attributes "{\"CrossZoneLoadBalancing\":{\"Enabled\":true}}"

Or

To enable cross-zone load balancing using the console

1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

2. On the navigation pane, under LOAD BALANCING, choose Load Balancers.

3. Select your load balancer.

4. On the Description tab, choose Change cross-zone load balancing setting.

5. On the Configure Cross-Zone Load Balancing page, select Enable.

6. Choose Save.

Answer 2

When you run nslookup dualstack.app.elb.us-east-2.amazonaws.com you are querying DNS and it's returning an IP address for an elastic load balancer node in each of the registered Availability Zones. Your client app then decides which IP to use for its request.

If you are requesting against the elastic load balancer node IP directly and not receiving a response, that means there are no targets responding to the node's request in that availability zone

https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/how-elastic-load-balancing-works.html#request-routing

Quoted for reference with formatting fixed

Routing algorithm

With Application Load Balancers, the load balancer node that receives the request uses the following process:

1. Evaluates the listener rules in priority order to determine which rule to apply.
2. Selects a target from the target group for the rule action, using the routing algorithm configured for the target group. The default routing algorithm is round robin. Routing is performed independently for each target group, even when a target is registered with multiple target groups.

With Network Load Balancers, the load balancer node that receives the connection uses the following process:

1. Selects a target from the target group for the default rule using a flow hash algorithm. It bases the algorithm on:

• The protocol
• The source IP address and source port
• The destination IP address and destination port
• The TCP sequence number

2. Routes each individual TCP connection to a single target for the life of the connection. The TCP connections from a client have different source ports and sequence numbers, and can be routed to different targets.

With Classic Load Balancers, the load balancer node that receives the request selects a registered instance as follows:

• Uses the round robin routing algorithm for TCP listeners

• Uses the least outstanding requests routing algorithm for HTTP and HTTPS listeners

Answer 3

This happens when you don't have any healthy target instance in one of the availability zones. Please check the target instances in both availability zones.