-1

In AWS, made 2 accounts.

  • account 1: A | Deployed EC2
  • account 2: B | Deployed RDS

Want to connect from account 1's EC2 to account 2's RDS, but doesn't work. So enabled VPC flow logs to diagnose where is the problem. The connection between these 2 accounts is using transit gateway. Both side have connected to self VPC with route table.

After did a connection test from A to B via db connect client, tried to find Network Interface ID from A, such as eni-XXXXXXXX belongs to the EC2 instance, in B's CloudWatch Logs Group, can't find its name.

I followed this guide to do, Using AWS VPC Flow Logs to Diagnose Intermittent RDS Connections. But these steps can't resolve my problem, I can't diagnose with it:

Determining your RDS Instance’s Elastic Network ID (ENI)

  1. Use the AWS RDS Console to obtain the instance’s endpoint (this will be in the form [xxx].[region].rds.amazonaws.com).
  2. Use dig or something similar to lookup the IP address of the endpoint. A CNAME is not sufficient, you’ll need the actual IP.
  3. Use the AWS Network Interface Console to search for the IP Address you got, and obtain the Network Interface ID (this will be eni-XXXXXXXX) from the resulting record.

Querying Your Flow Logs

  1. Use the VPC Console to locate your VPC. Go to the Flow Log tab, and click through to the CloudWatch Logs Group.
  2. In the CloudWatch Logs Group, use the search box to find your Network Interface ID.
  3. Click into the record you’ve found, and you’ll see all the traffic for your instance!

How can confirm the network traffic each other correctly?

I found an unknown port in the flow logs.

2 023910321035 eni-12309fdsaj0jf012 ACCOUNT_1_EC2_IP ACCOUNT_2_RDS_IP 45052 31337 6 3 180 1619490209 1619490213 REJECT OK
2 023910321035 eni-12309fdsaj0jf012 ACCOUNT_1_EC2_IP ACCOUNT_2_RDS_IP 45052 31337 6 3 180 1619490209 1619490213 REJECT OK

The port 31337 didn't open. Why it will connect this port?

Miantian
  • 137
  • 7
  • 1
    I'd deploy an EC2 in account 2, open up ICMP / ssh, and test with ping / ssh. Check your VPC and transit gateway route tables, security groups on both ends, NACLs. – Tim Apr 27 '21 at 01:35
  • @Tim I checked everything you mentioned above. All of them look good. I also tested in account 2 with an EC2 instance. It can connect to the RDS in account 2. – Miantian Apr 27 '21 at 02:47
  • @Tim And we aren't using `transit gateway route tables` but normal route tables. Other accounts work fine. They are using the same configuration method. – Miantian Apr 27 '21 at 03:03
  • You need to configure VPC route tables to allow the VPCs to communicate with Transit Gateway, and Transit Gateway route tables to tell it where to route data. Depending on your TG configuration the TG route table may be populated automatically from VPC route propagation but you need to check it. – Tim Apr 27 '21 at 03:48
  • @Tim Thank you. I confirmed VPC route tables in multiple accounts. They look fine. And now I can get the flow logs from account 1 to account 2. I updated the question and added some data there. Just not clear why `31337` will be pointed. – Miantian Apr 27 '21 at 04:28
  • Could be ephemeral ports ( https://en.wikipedia.org/wiki/Ephemeral_port ). Suggest you follow my earlier advice, set up EC2 instances and allow SSH / IMCP and trace a known transaction. Ignore the rest, networks can be very noisy and flow logs can take a lot of effort to follow. – Tim Apr 27 '21 at 07:30
  • @Tim How to use `IMCP` on EC2 in account 2? – Miantian Apr 27 '21 at 08:32
  • 1
    Just ensure ICMP can make it through security groups, AL2 and Ubuntu respond to ping by default. Given the length of the comments here maybe you're better off getting a consultant in to help you. This kind of thing can be fiddly, sometimes some experience means it's done in 2 hours instead of 2 weeks. – Tim Apr 27 '21 at 08:57
  • @Tim Thank you very much. – Miantian Apr 27 '21 at 09:03

1 Answers1

1

  1. Do you really need Transit Gateway? As suggested in the answer to your other question VPC Peering may be enough. If not tell us why.

  2. In that other question you confirmed that your setup in the end worked and you could connect from EC2 to RDS. If you can't now what has changed?

  3. As suggested in the other answer and also by @Tim here in the comment do this:

    • Spin up an EC2 instance in the same Subnet and Security Group as your RDS.
    • Use netcat to listen on the DB port (e.g. 3306 for mysql) and tcpdump to observe the traffic from the "other" EC2.

    Using that figure out if your connectivity works - do you see the incoming packets with tcpdump? Do you see the responses?

MLu
  • 23,798
  • 5
  • 54
  • 81
  • Thank you again. Yes, I confirmed the connection between multiple AWS accounts before. Actually we have many accounts, most of them can connect to the RDS but only one can't. In order to verify where is the trouble, we enabled VPC Flow Logs to check(Thanks to you for the suggestion). And we already designed our network solution with Transit Gateway so we can't change it. I can confirm that the connection from account 2's EC2 is available. But can't from account 1. I tried to use `netcat` and `tcpdump` to get something but nothing happened(Added to the bottom of the question). – Miantian Apr 27 '21 at 02:54