3

I am using two public subnets in two availability zones, and then two private subnets in those same availability zones. I have an elastic load balancer that forwards traffic to my EC2 instances in the private subnets.

I have a bastion server in the public subnet and that is how I can reach the instances in the private subnets.

I can SSH to the bastion server, SSH into the servers from there on the private subnets, and on those servers I can successfully run sudo yum update, my load balancer can talk to them, etc, BUT I cannot git pull from Bitbucket (it just hangs)...

I even tried updating the Network ACL on the private subnets to allow all traffic and that didn't work either, so the issue appears to not be the Network ACL.

EDIT: I forgot to mention that I have NAT Gateways configured and I can make HTTP and HTTPS requests to the outside world from my private subnet...

EDIT (screenshots of route tables):

Private table 1: Private table 1

Private table 2: Private table 2

Public table: Public table

EDIT (Network ACLs):

Private inbound: Private inbound Network ACL

Private outbound: Private outbound Network ACL

Public inbound: Public inbound Network ACL

Public outbound: Public outbound Network ACL

Any ideas why I can't git pull?

lorenzo
  • 31
  • 5
  • 1
    Can you post a screenshot of your private subnet route table? I suspect it’s got something to do with routing and/or NAT. – MLu Nov 13 '18 at 20:30
  • @MLu screenshots of route tables added – lorenzo Nov 13 '18 at 20:41
  • 1
    Thx for the screenshots, that looks ok. Can you verify the NAT gateways are in the public subnets? I guess they are if you can run yum. What about the *outbound rules in the security group* of the instance? Do they permit access to `0.0.0.0/0`? And from that instance can you `ping 8.8.4.4`? – MLu Nov 13 '18 at 20:52
  • @MLu I just verified that both NAT gateways live in the public subnets. My security groups on the instances allow outbound to 0.0.0.0/0. I can't ping 8.8.4.4 though.... – lorenzo Nov 13 '18 at 20:58
  • 1
    That's weird. Does the instance have a *Public IP* by any chance? That may interfere with NAT. Make sure it doesn't have Public or Elastic IP attached. And for NACL - do they have any specific inbound or outbound rules? – MLu Nov 13 '18 at 21:04
  • @MLu the instances inside the private subnet do not have public IP addresses. I just posted screenshots of the Network ACLs – lorenzo Nov 13 '18 at 21:17
  • 1
    What's the GIT URL you're trying to pull from? Does it resolve to a private IP by any chance? What protocol does it use? https://? git://? ssh://? What's the hostname? – MLu Nov 13 '18 at 21:41
  • @MLu it was using SSH to get to bitbucket DOT com. I finally figured out the problem (answer posted). Thanks for the help! – lorenzo Nov 13 '18 at 21:50
  • 1
    Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/85712/discussion-between-mlu-and-lorenzo). – MLu Nov 13 '18 at 21:54

2 Answers2

2

What is in your Private subnet Route Table? I strongly suspect that you don’t have a NAT Gateway configured and route to the Internet set through the NAT.

See here for more info about why may you need a NAT Gateway for private subnets: NAT gateway for EC2 instances

Hope that helps :)

MLu
  • 23,798
  • 5
  • 54
  • 81
  • My public subnet route tables use the internet gateway. My private subnet route tables use the nat gateway. Inside the private instances I can make HTTP and HTTPS requests but cannot git pull using SSH, nor can I do any other SSHing – lorenzo Nov 13 '18 at 18:56
  • 1
    @lorenzo What about the *Private subnet*? How do the hosts from there get to the internet? They need NAT. Updated the answer with a link. – MLu Nov 13 '18 at 18:58
  • my private subnet uses the NAT gateways which seem to be working fine since I can make HTTP and HTTPS requests to the outside world... – lorenzo Nov 13 '18 at 19:59
0

The underlying issue was that the network traffic was making it out of the private subnet and not into the public subnet (where it was supposed to depart and go to the public internet). I had to update the SSH rules in the Network ACL on the public subnet.

lorenzo
  • 31
  • 5