1

I'm trying to setup a lambda function to process logs from an AWS instance. To do that I'm configuring a CloudWatch Logs "trigger" as input to my Lambda function. The relevant configuration UI on AWS console is shown in the screen shot below for reference.

I have a log group and in that group is a bunch of log streams, each corresponding to a log file. I only want to process log events from 2 of +30 streams in the log group. The CloudWatch Logs Lambda input trigger allows you to specify a log group and a filter. Oddly, there doesn't seem to be an option to filter logs based on the stream name. I can just filter all logs in my lambda function but I would rather do the work before my lambda is executed otherwise it's being invoked with irrelevant logs 99% of the time.

Regarding the filter parameter, the Lambda CloudWatch Log integrations docs points to Filter and Pattern Syntax for the filter. But I can't see anything in there about log stream names.

enter image description here

spinkus
  • 158
  • 1
  • 15

1 Answers1

1

The filter doesn't work like what you want. You can't filter based on log stream, only on log line content.

I suggest you have no filter (eg so the Lambda receives all log events) and check the log stream in the event metadata when your Lambda is invoked. Details on the event format are in the Lambda docs:

https://docs.aws.amazon.com/lambda/latest/dg/services-cloudwatchlogs.html

Alex J
  • 2,804
  • 2
  • 21
  • 24
  • That is my current solution. The issue is the files I'm interested are seldom updated, like every few minutes, where as other files in the log group are updated in the order of every second or less, so means for every invocation that actually results in useful work being done, my lambda is getting invoked 100s if not 1000s of times just do nothing. – spinkus Oct 25 '19 at 07:36
  • https://docs.aws.amazon.com/en_us/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData-discoverable-fields.html says you can filter by `@logStream` – jrc May 27 '22 at 14:36