EC2 instances can't reach each other

Question

I created two EC2 instances in the same AZ and on the same account. They use different security groups. I'd like instance A to accept connections on a certain port from only instance B.

I don't believe these instances are VPC, but don't know how to confirm. I wasn't able to change the security group which makes me think they are not VPC.

In the security group for instance A I added a rule for the port and used instance B's public IP /32 for the source. I then tried to connect from instance B using instance A's public IP, but the connection attempt fails immediately.

I tried the same steps with the private IP of each instance. What am I missing?

Here's an article which answers a similar question, but VPC is involved: Can't connect to EC2 instance in VPC (Amazon AWS).

Both instances have the same VPC ID and Subnet ID.

I also tried setting the source to instance B's security group, which didn't work either.

I'm trying this with mysql. The mysql client running on instance B failed immediately with this error:

ERROR 2003 (HY000): Can't connect to MySQL server on '54.xx.xx.xx' (113)

To check there wasn't a problem with mysqld setup, I tried the same with ICMP Echo Reply which didn't work either.

Edit Thanks to initial answers I was able to confirm these two instances are running in a VPC (by going to the VPC console). So, my question is very similar to the linked article. But, in that case the problem was that the instances were not default instances so didn't have the proper route and subnet created. Here's how my VPC is set up: The VPC is default and has a route table associated with it. The route table is implicitly associated with the subnet associated with the VPC. The route table has a single route in it and the target is "local".

These are all created by default as as I understand the docs should allow two instances to connect to each other. What am I (still) missing?

Answer 1

I resolved this with help from AWS tech support. Here's the info for future newbie's like me:

The issue was that iptables was running on instance B and not allowing any traffic. I learned that there are two levels of firewall for EC2 instances: security groups (managed at the AWS console) and iptables (managed on the host). There are reasons to use iptables, for example https://wincent.com/wiki/Using_iptables_on_EC2_instances

Most of the time you don't need to worry about using a host-level firewall such as iptables when running Amazon EC2, because Amazon allows you to run instances inside a "security group", which is effectively a firewall policy that you use to specify which connections from the outside world should be allowed to reach the instance. However, this is a "whitelist" approach, and it is not straightforward to use it for "blacklisting" purposes on a running instance.

In my case I don't need host level firewall so turned iptables off:

sudo service chkconfig stop
sudo chkconfig iptables off

Here are some results I got related to the comments posted on this question:

• connecting with private ip worked
• connecting with private DNS name worked
• connecting with public ip worked
• connecting with public EIP worked
• connecting with public DNS worked, but as Chad Smith said in his answer DNS returns the private IP for this name

The reason this worked for me on a different instance is that the image I used in that instance didn't run iptables -- every image is different. The image I used in this case used iptables to disallow all connections except SSH.

Answer 2

A little bit off topic, but this is the only search result for this issue.

We had a similar problem, but our existing instances were rebooted and suddenly couldn't communicate. Turns out there were too many rules in the security group - just removing some allowed communication to resume. It was still working before the reboot because the rules get added over time by automated calls to the api.

Hope this helps someone in the future.

Answer 3

If you cannot modify the security settings of the running instances, they are NOT launched into a VPC.

Even for instances not in VPC, EC2 launches them into private networks which are interconnected. So you should specify the private IP address of instance B in the security group of instance A.

Answer 4

(1) you can check to see if your instance is in a VPC via the AWS console. In the EC2 dashboard, you can select your instance, then in the Description tab, on the left hand column, there is a VPC ID field. If this is blank, you are in EC2-classic.

(2) You cannot access the public IP of an instance from within another instance in EC2 unless that port is open to the world. Your error above using IP 54.X.X.X tells me you are using the public IP. Change your connection string to use the public DNS instead. This will be the hostname that starts with ec2-. When you do a DNS lookup on that public dns name, it will resolve to the PRIVATE IP instead of the public IP, which should be reachable if your security groups are set up correctly.

summary: try connecting to your mysql instance using the public DNS of the instance.

If you still cannot connect, verify that mysql is listening on eth0 and not just on the loopback interface.

Answer 5

AWS has separate security groups for VPC and non-VPC instances, so you need to somehow find out if you are on VPC or not (just go to VPC console and check if you see your instances there) and then make sure security groups you have created are in the same context. Then you can just add security Group A to Security group B as trusted and do the opposite too. This way you just allow all traffic between two hosts on an individual ports (I assume that this was your intention).