6

On my AWS Lambda function, my javascript code times out whenever I try to use nodemailer to connect to my Amazon SES SMTP server (port 465). However, if I run the script locally, it works fine, which leads me to believe it's either a problem with the lambda dialing out to the SMTP server, or the SMTP server blocking the lambda from connecting -- I suspect the former is the issue.

I am using a firewall behind my Cloudfront distribution, but I don't think this is applied to incoming SES connections or outgoing lambda functions. In VPC, I can see there is an Internet Gateway attached to the instance. The outgoing connections for the Security Group allows all protocols to access 0.0.0.0/0, however, the ACL looks odd in that it's both allowing and rejecting all inbound/outbound connections:

enter image description here

enter image description here

In VPC, I see 6 subnets listed, where it's not very obvious to me what exactly these are doing in the grand scheme of things.

enter image description here

In the logs, I just see Task timed out after 6.01 seconds

Any idea how I can get more information on where the hangup is at?

iRyanBell
  • 413
  • 2
  • 11
  • 20

1 Answers1

12

This is expected.

Lambda functions in a VPC can't communicate with the Internet (including the standard service APIs) using an Internet Gateway, because an Internet Gateway requires the internal devices to have associated public IP addresses. Being on a public subnet (where the default route is the Internet Gateway) isn't sufficient.

Important

If your Lambda function needs Internet access, do not attach it to a public subnet or to a private subnet without Internet access. Instead, attach it only to private subnets with Internet access through a NAT instance or an Amazon VPC NAT gateway.

https://docs.aws.amazon.com/lambda/latest/dg/vpc.html

A NAT device -- typically a NAT Gateway -- is required, unless the service in question supports VPC Endpoints (which SES currently does not).

Place the NAT Gateway on a public subnet (so that it can access the Internet using the Internet Gateway) and then create one or more private subnets, pointing their default route to the NAT Gateway.

The NAT Gateway is the newer alternative to the NAT Instance, which is an EC2 instance dedicated to the same purpose. This was formerly the only way to privide the required NAT service. Unlike a NAT Gateway, which is managed by AWS and is fault-tolerant, a NAT Instance represents a potential single point of failure (but has a lower associated cost).

Or, you can move the Lambda function out of the VPC if it requires no other VPC resources.

The Network ACL both allowing all and denying all is normal, because rules are processed in order. That last rule is the default behavior that would apply if the Allow rule is removed. It's mostly a visual cue to remind you why the NACL doesn't work if you delete the other rules. Users might otherwise assume that since they didn't explicitly deny something, that it should be allowed.

Each network ACL also includes a rule whose rule number is an asterisk. This rule ensures that if a packet doesn't match any of the other numbered rules, it's denied. You can't modify or remove this rule.

https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html

Michael - sqlbot
  • 21,988
  • 1
  • 57
  • 81
  • Ok, I created an NAT Gateway. I'm not quite seeing how I get my lambda function to use this to connect to the outside internet. It looks like this is done through some sort of routing table route propagation (through an "internet gateway") based on subnet addressing? I suppose it's not as simple as attaching the subnet to the VPC and then attaching the NAT gateway to the subnet (do I need all of these subnets?) I'm feeling a little bit like the meme of the dog with the test tubes. – iRyanBell Feb 24 '19 at 01:16
  • Create the NAT Gateway on one of the *existing* subnets. It needs to be located on a subnet with the default route pointing to the Internet Gateway so it can access the Internet. Then, create a new route table, with the default route pointing to the NAT Gateway. Then, create two new subnets, in two availability zones, using this new route table for those subnets. Attach the Lambda function to the new subnets. – Michael - sqlbot Feb 24 '19 at 01:40
  • Hmm, now I get `Task timed out after 6.01 seconds` when my lambda tries to connect to Elasticache. The lambda, database, ACL, and NAT Gateway are all on the same subnet, and the route table has the internet gateway attached. Why would I need to create new subnets... and why different availability zones than my servers? (and what would I use for "IPv4 CIDR block") ? – iRyanBell Feb 24 '19 at 01:54
  • 1
    You need different subnets because, as I mentioned above, you have to *"Place the NAT Gateway on a public subnet (so that it can access the Internet using the Internet Gateway) and then create one or more private subnets, pointing their default route to the NAT Gateway."* A NAT Gateway can't be on a the same subnet it serves. If all your resources are in a single AZ, you can just use one new subnet in that AZ. Based on your current subnets, it looks like you could use CIDR block 172.31.x.0/20 where x is 96, 112, 128, 144, any number up to 240 as long as it is divisible by 16. – Michael - sqlbot Feb 24 '19 at 02:10
  • I appreciate you taking the time to help me figure this out. Here's the setup, with the lambdas using both of the private subnets. My elasticache is back in business, but I'm still not getting the SMTP connectivity. I believe the Routing Table and its Subnet Associations needs a different configuration, but I'm not quite following what it should look like. https://i.imgur.com/LO6tkeJ.png – iRyanBell Feb 24 '19 at 02:22
  • 1
    Associate the new route table with the new subnets -- not the subnet the NAT Gateway is on. The new route table is tied to the NAT Gateway because its 0.0.0.0 route needs to specify the NAT Gateway as its target. Also note that 172.32.x.x and 172.33.x.x are not valid CIDR choices. They are allowed because the system doesn't enforce RFC-1918 compliance, but those are definitely not valid choices, here. If the IP address starts with 172, the second number must be between 16 and 31. See my suggested blocks, in the previous comment. – Michael - sqlbot Feb 24 '19 at 02:36
  • Ah, it works! What a crazy little setup. – iRyanBell Feb 24 '19 at 03:45
  • Amazon Simple Email Service (Amazon SES) is now supported via Interface endpoints (powered by AWS PrivateLink): ["Setting up VPC endpoints with Amazon SES "](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-set-up-vpc-endpoints.html), but I'm not sure if it works with Lambda. The documentation only mentions EC2. – Carl G Jul 27 '20 at 15:16