1

I've set up Amazon SES to send a company announcement to a list of about 1,000 contacts. I set up the Kenesis Firehose to log all e-mail events (e.g., Send, Bounce, Click) to a bucket in S3. SES seems to provide tools for bulk analysis of massive numbers of e-mails, but I want to see results for each individual recipient. I don't know if I missed something, but the only way I found to do that was to download the files from S3 and parse them in a spreadsheet. I've developed a pretty complex spreadsheet to do that.

The files are stored in S3 in a folder hierarchy by month, day, and hour. The S3 console allows me to download each file individually by navigating through the folder tree manually and right-clicking on each file. The S3 console documentation says:

You can download a single object per request using the Amazon S3 console. To download multiple objects, use the AWS CLI, AWS SDKs, or REST API.

I've become familiar with the AWS SDK for PHP and am using it to send the e-mails in SES. The S3 Developer Guide has instructions to Get an Object Using the AWS SDK for PHP. It doesn't seem to have instructions for getting multiple objects, and I guess I can accomplish that by writing a loop through the folders and files.

I have not installed the AWS CLI. There's a Server Fault answer that seems to say that one can download a folder by the CLI command sync.

So what it looks like right now is that in order to download all the files in a folder, I either need to write an SDK program or install the CLI and learn the sync command. Either one of those seems like a lot of work for something I can do in Windows with a mouse drag or in Filezilla with a double click of the mouse. Am I missing something, or do I really need to do all that work just to download the files in a folder tree?

Windows and Filezilla also easily allow me to see a whole folder tree at once with all the files in each folder. The S3 console only allows me to see one subfolder at a time. Again, do I need to write an SDK program or learn the CLI just to get a listing of a folder tree?

While I'm asking those two questions, something else that would be helpful would be to see the number of lines in each file because each line represents an SES event. I'll have that information easily from my analysis after I have the files, but I'm surprised that SES doesn't seem to give me a way to see the number of events except by doing that analysis. Is that correct, or have I overlooked something in SES that would give me that information?

And one final question: All of the above would be unnecessary if I could simply ask SES to give me a dump of all the event data. The only way I've found to get those data is by downloading those S3 files, which I then have to merge into my spreadsheet. So again, have I overlooked something in SES that would allow me to get all event data without going through all this rigmarole in S3?

NewSites
  • 113
  • 3

1 Answers1

2

You can simply use cyberduck to list and download files, or use Athena to analyze your data directly from S3

ehmad11
  • 138
  • 6
  • 1
    Re Cyberduck: Wow! Thank you. An "FTP" client that connects to S3, even though S3 doesn't run FTP. It's perfect. It creates a seamless GUI interface between Windows File Explorer and S3. (Not actually, perfect though: The file count in the status bar at the bottom does not distinguish folders and files and counts them all as files. A nuisance I can live with.) – NewSites Jun 03 '20 at 19:24
  • 1
    Re Athena: Thanks for that suggestion, but I think it's easier to do the analysis I want locally in Excel. For reference of others, he was talking about Amazon Athena, an AWS tool for running SQL queries on data in S3: https://aws.amazon.com/athena/ , additional description at https://aws.amazon.com/blogs/big-data/analyzing-data-in-s3-using-amazon-athena/ . – NewSites Jun 03 '20 at 19:31
  • It's really interesting that Amazon doesn't either provide a GUI for S3, nor say in its documentation that one is available in Cyberduck. I'm really impressed with AWS's detailed (though not flawless) documentation. They obviously have put a lot of resources into that, so it's surprising that they don't go the extra step of a GUI, which would be easy for them to do. I wonder if Amazon is designing AWS to be a professional toolbox, only suitable for geeks, so they provide good, professional-caliber documentation, but no GUI. – NewSites Jun 03 '20 at 19:41
  • i believe S3, EC2 etc are just fundamentals, services like Cloud​Formation are the future – ehmad11 Jun 03 '20 at 19:50
  • I don't know either EC2 or CloudFormation, but I looked up the latter (http://amazonaws.cn/en/cloudformation), which is subtitled "Model and provision all your cloud infrastructure resources." Does it use something other than S3 for storage? – NewSites Jun 04 '20 at 00:38