0

I have noticed a weird behavior regarding POST requests on all my servers from devices that are connected to the internet either via 4G or via hotspot from a 4G network.

I am an admin of 2 services hosted at AWS. Both services use the same Elastic Load Balancer and are hosted via ECS (docker). Both services run with gunicorn and django but listen on different endpoints.

The thing is that ALL of the POST requests made from a device with a 4G or a hotspot connection never ever seem to even reach the ELB (according to the logs no request is recorded), which is the first entry point that I control.

I used debug mode and tried some requests from my Android device and I can see from the Chrome dev tools Network tab that the request silently fails with no response code. It just stalls.

I can freely copy the request as curl and it works.

The JS code is nothing fancy, a typical AJAX POST request.

Any ideas as to what is going on? Any tips as to how to debug this?

Nikolaos Paschos
  • 123
  • 1
  • 7
  • What do the ELB access logs say? – Tim Aug 22 '19 at 18:38
  • They do not even show a trace of the requests. No relevant record whatsoever. – Nikolaos Paschos Aug 22 '19 at 18:44
  • I guess the request doesn't arrive at the ELB, or maybe it's malformed and discarded. Can you do something like set up a commercial VPN on that network? Just curious what would happen. – Tim Aug 22 '19 at 19:13
  • Is your ELB configured for IPv6? – Michael Hampton Aug 22 '19 at 19:37
  • I cannot set a VPN but I can follow the stream with wireshark. I will followup soon. My application ELB is configured as dualstack and my subnets have IPv6 CIDR blocks. @MichaelHampton – Nikolaos Paschos Aug 22 '19 at 19:43
  • Check your DNS records, then, and make sure they point correctly to your ELB for both IPv4 and IPv6. – Michael Hampton Aug 22 '19 at 19:44
  • My DNS (hosted at namecheap) has 2 records (one for each service) that both point to my ELB. The records are CNAME because Amazon does not reveal the IP of ELBs but rather provides a name and internally handles the details with Route 53. I cannot somehow find the actual IP to point my records. – Nikolaos Paschos Aug 22 '19 at 19:51
  • I used http://ipv6-test.com/validate.php to check whether my services are accessible via IPv6 and they are not. The AAA records are OK but then IPv6 web serves time out. Do you think that is the issue? Still, no records on ELB logs though @MichaelHampton – Nikolaos Paschos Aug 22 '19 at 20:07
  • Yeah,, that's most likely it. Remember that 4G LTE is an IPv6 based system, and most mobile data goes over IPv6 (unless the carrier is significantly doing it wrong). – Michael Hampton Aug 22 '19 at 20:39
  • I will try solving it tomorrow then and I will update on the situation as soon as I know more. Until then, thank you for the ideas, I was stuck. – Nikolaos Paschos Aug 22 '19 at 21:09

1 Answers1

2

I will post the update here so anyone in the future can clearly see the solution.

As suggested by Michael Hampton the problem was that my stack was not configured properly to handle IPv6 connections. To be precise; My ELB was already in dualstack mode, my VPC had 2 subnets which both already had CIDR blocks associated with IPv6 BUT my routing table did not have a routing rule for ::/0 and thus all traffic was from IPv6 was dropped. I forwarded all of the IPv6 traffic to an internet gateway associated with my VPC and now everything runs smoothly.

Nikolaos Paschos
  • 123
  • 1
  • 7