How to verify a AWS VPC (S3) endpoint works?

Question

I added a VPC endpoint to my VPC using CloudFormation, and allowed s3 usage. The routes are visible in the AWS console, but not in the local routing tables of the EC2 instances:

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.29.4.129    0.0.0.0         UG    0      0        0 eth0
169.254.169.254 0.0.0.0         255.255.255.255 UH    0      0        0 eth0
172.29.4.128    0.0.0.0         255.255.255.128 U     0      0        0 eth0

How do I verify that the EC2 instances in the VPC actually uses the VPC endpoint for S3, and not the available internet connection?

Answer 1

I guess the straightfwd way is to actually probe those routes.

You can traceroute to s3 and see if the NAT Gateway's internal IP is anywhere in the output (eg. the first hop).

First, check the NAT Gateway internal IPs in the console.

Example output with the endpoint set - no gateway IP shown. This is what you want to see.

$ traceroute -n -T -p 443 s3.amazonaws.com                                
traceroute to s3.amazonaws.com (52.216.204.93), 30 hops max, 60 byte packets
 1  * * *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  52.216.204.93  0.662 ms  0.668 ms  0.637 ms

Example output of a different destination, going via NAT (see the first hop)

$ traceroute -n -T -p 443 serverfault.com
traceroute to serverfault.com (151.101.129.69), 30 hops max, 60 byte packets
 1  172.20.10.188  0.206 ms  0.147 ms  0.145 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  100.65.13.49  0.956 ms 100.65.13.113  1.253 ms *
 8  52.93.28.209  1.083 ms 52.93.28.231  1.213 ms 52.93.28.235  1.151 ms
 9  100.100.4.38  1.770 ms 100.100.4.46  2.089 ms 100.100.4.36  1.723 ms
10  103.244.50.242  1.136 ms 100.100.4.44  1.702 ms  2.738 ms
11  151.101.129.69  1.013 ms 103.244.50.244  1.745 ms 151.101.129.69  1.142 ms

Answer 2

I have found a method to verify the VPC endpoint usage.

1. Log in to an AWS EC2 instance in the VPC
2. Configure the aws cli client
3. run aws ec2 describe-prefix-lists; for Windows PowerShell, Get-EC2PrefixList

The result should contain the the VPC endpoints prefix list ID in the attribute PrefixListId.

For additional verification, you can apply the following policy to an S3 bucket:

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Principal": "*",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::mybucket"
      ],
      "Condition": {
        "StringNotEquals": {
          "aws:sourceVpc": [
            "vpc-121212"
          ]
        }
      }
    }
  ]
}

with your vpc ID instead of vpc-121212. You should then only be able to access the S3 bucket from the given VPC

Answer 3

You can turn on S3 logging and check if the files are being accessed from your private IP rather than public. If your logging shows private IPs are accessing the buckets you've configured it correctly. Goodluck!

Answer 4

I would recommend to launch ec2 instance (with IAM role allowed to list s3 buckets) in subnet without internet access.

Basically only 2 active rules in route table (your VPC subnet range and s3 endpoint).

Connect to instance and run command:

aws s3 ls /**

It should fail with timeout because boto by default will create request to global s3 url (s3.amazonaws.com).

export AWS_DEFAULT_REGION=us-east-1** ## your region here
aws s3 ls /**

should list your buckets in us-east-1 region (vpc router will route your request to s3.us-east-1.amazonaws.com).

Answer 5

@m-glatki's answer (which for some reason is the accepted one) is factually incorrect.

First of all, you have to explicitly enable an ec2 VPC interface to even be able to perform the aws ec2 describe-prefix-lists call, otherwise you will get a timeout.

Secondly, even if you can call that api, it won't tell you whether you are routing your traffic through that endpoint. It just provides details about a specific prefix list (PL) in the current region.

What you have to do is associate an S3 VPC endpoint to the subnet's route table and make sure your EC2 instance or service's security group allows egress connectivity to via that endpoint (you should be fine with the default "allow all" egress rule). This will route S3 traffic via the endpoint, even if you have a NAT gateway attached to it.

You can verify that your traffic is not routed through the NAT by checking its associated cloudwatch logs (See BytesOutToDestination, BytesOutToSource, BytesInFromDestination, and BytesInFromSource metrics)

Also check S3 bucket logs as @michael correctly pointed out.

Answer 6

Elaborating on @m-glatki solution, add a policy on the bucket that restricts S3 access to a particular VPC Endpoint:

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::mybucket",
            "Condition": {
                "StringNotEquals": {
                    "aws:SourceVpce": "vpce-01ab2c3d4"
                }
            }
        }
    ]
}

You will only be able to list bucket contents from a process that uses the VPC endpoint. Otherwise, you will receive a message:

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

To get the VPC Endpoint ID, use this command: aws ec2 describe-vpc-endpoints

See this link

Answer 7

Your instance forwards packets destined to S3 to the local gateway, and from there the VPC 'router' forwards them to the S3 endpoint. No client configuration or knowledge is required.

You could configure the S3 endpoint with a very restrictive set of ACLs such that it denies all requests and observe your client receive the failure as well.