11

I was wondering if I can send a message to SQS queue and subscribe an SNS topic to it to trigger a lambda for sending an email.

SQS -> SNS -> (Lambda) -> SES

I know SNS messages can be sent to SQS but I'm curious if the other way around is possible

spro
  • 103
  • 2
Chida
  • 2,471
  • 1
  • 16
  • 29

4 Answers4

11

One thing I did was to create a CloudWatch alarm on ApproximateNumberOfMessagesVisible (>= 1 for 5 minutes) for the SQS queue. The alarm publishes to an SNS topic that triggers the lambda function. The lambda function loops until it clears the queue.

It can take up to 5 minutes to trigger from the alarm, but it works fantastically for batch-scheduled tasks without needing to poll the queue. (Alarm granularity is 5 minutes for active queues.)

squidpickles
  • 751
  • 1
  • 8
  • 12
  • I'm thinking about doing this exact same thing. Do you know if you get charged each time cloudwatch checks your queue? Not that we're really talking big bucks with this sort of thing just more curious. – Brian F Leighty Mar 23 '16 at 13:58
  • Also, I'm assuming from what you say the alarm keeps sending SNS of the alarm over and over? What do you have the timeout on your lambda function set to? – Brian F Leighty Mar 23 '16 at 14:00
  • One other thing. Sorry for all the comments. I was a little worried with the "Approximate" part have you ever had a time where there was a message waiting there that didn't get dealt with since it thought there was only 0 items instead of the 1 item there? – Brian F Leighty Mar 23 '16 at 14:02
  • @BrianFLeighty the checks are free. But it doesn't alarm continuously; it may require a second alarm for messages sitting on the queue (added between lambda execution completion and next alarm). Polling may be a better solution; mine works for a typically very quiet queue (hence not worth polling.) – squidpickles Mar 23 '16 at 20:18
  • Missed the "approximate" comment. That seems to work fine, but ymmv – squidpickles Mar 23 '16 at 21:48
7

You can't go SQS -> SNS, only SNS -> SQS.

Lambda now supports scheduling so one option is to implement an SQS poller in a Lambda function and run it frequently.

Another option to consider is whether you actually need a queue. Lambda supports asynchronous processing (via the Event invocation mode) and should transparently scale horizontally to handle parallel invocations. If your lambda function doesn't require access to a central state store which might constrain parallel execution then you could probably just run all your invocations in parallel. I believe there's a 100 concurrent execution limit per account though, so you may need to batch your messages to stay under that.

thexacre
  • 1,849
  • 12
  • 14
  • 3
    there is also nothing wrong with using traditional queue poller tricks with lambda: e.g. if lambda dequeues a message, during execution, then retrigger the function at the end; otherwise let it execute next as scheduled – nik.shornikov Apr 26 '16 at 01:35
1

SQS queue can be subscribed to SNS topic and so to process received SNS messages. Currently, it is not doable in other direction without additional coding (see e.g. Lambda FAQ).

I would say there is a couple of options how to do it but it is not so elegant as using more common event-driven system AWS event->SQS->Lambda. Otherwise you may need to customize/implement the code how SQS queues are processed:

  1. you can implement your own event sources
  2. you can have some intermediate EC2 instance to listen to SQS queues and then to trigger Lambda on SQS events
dsmsk80
  • 5,757
  • 17
  • 22
0

This was asked and answered a while ago, but having just thought about this myself, I thought I'd add an approach.

As mentioned, Event Sources may be the best bet here. Alternatively, and I haven't tested this nor thought this through (so this is kind of academic), but it may be possible to accomplish this via a Fan-Out pattern with SNS as follows:

 1. Create a SNS topic.............................: SNS-topic-01
 2. Subscribe a SQS queue to that topic............: SQS-queue-01
 3. Subscribe a Lambda Function to that topic......: LAMBDA-func-01

Using this configuration, submitting a message to the SNS topic will enqueue it to the SQS queue while simultaneously triggering a companion Lambda function. That Lambda function would be written to read that very same SQS queue but with Long Polling enabled (up to 20 seconds) so that it doesn't read the queue before the enqueue completes (i.e. race condition).

In essence, this scheme just-in-time invokes one Lambda function for each enqueued SQS message. I don't know how simultaneous Long Poll readers work on SQS (... does one get dropped?), but this is just another way to consider solving this. =:)

NYCeyes
  • 111
  • 1
  • 5