I have a list of users in .passwd and access_log being uploaded to Cloudwatch.
I need to make weekly report who from this list did access to server.
Is there any way to automate this rather than checking logs manually for each user? There are several dozens of them.
To automate a task like this you've got a couple of options.
Batch processing - Download logs from CloudWatch and then parse them, looking for your users.
You can use aws logs describe-log-streams to find the log stream names, filter it by timestamp to make sure you're only processing the recent ones.
For each log stream call aws logs get-log-events which will give you the actual log messages and then parse them, e.g. using grep.
Instead of using aws and grep you may want to write a small downloader / parser e.g. in python or in some other language of your choice and use one of the AWS SDK libraries for accessing the logs.
Online processing - capture and process the matching log records as soon as they are written to CloudWatch Logs. You can set up CloudWatch Logs Filter pattern and feed the matching access_log records e.g. to a Lambda function that will keep track of the users accessing your website. You can keep the results for example in a DynamoDB, one object for each user.
Go heavy weight and use ElasticSearch Service or Graylog or Splunk or some similar log processing service. But that's very likely an overkill for your use case.
Hope that helps :)
