0

What I want to achieve is to scale the number of consumers of a defined queue using lambda functions, instead of "self-hosted" subscribers.

For a given event-queue I want several consumers processing unique messages from this same queue, just like I would simply do with a cluster of processes, but I would like to use the cloud-hosted lambda functions. So far I just found how to subscribe with a single function, without scaling option.

joaoricardo000
  • 101
  • 3
  • Have you considered one Lambda function that fires off several others, either directly via Lambda or via a set of separate SQS queues? – ceejayoz Mar 14 '18 at 21:06
  • Yes, but it would not be as simple to scale as it would be with just managing the number of active subscribers. – joaoricardo000 Mar 14 '18 at 21:15

1 Answers1

2

I found this very helpful article from cloudonaut.io (by Andreas Wittig), that exemplify one approach to multiple AWS Lambdas consumers:

Integrate SQS and Lambda: serverless architecture for asynchronous workloads

To scale Andreas' solution you can use AWS CloudWatch Alarms from your AWS SQS queue and tigger AWS CloudFormation (and/or AWS Serverless Application Repository) to spin up more AWS Lambda consumers when needed.

Another approach is to use Amazon MQ.

If you use Amazon Simple Notification Service (SNS) or AWS Kinesis, the same message will trigger different lambdas and you'll probably get duplicated actions between your lambdas.

[update]

Now it is possible to use Amazon Simple Queue Service as a trigger for AWS Lambda:

More info here: https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/

Lucas Carvalho
  • 531
  • 4
  • 10