AWS S3 CLI: sum bytes in all top-level prefixes with a single --query?

2

I have an S3 bucket with a bunch of top-level prefixes. I have a relatively inefficient way of summing the bytes across top-level prefixes:

  1. Get the list of prefixes via aws s3 ls [bucket name], followed by some sed/grep.
  2. do a bash for-loop over those prefixes, running aws s3api list-objects --bucket [bucket name] --prefix $prefix --output json --query [sum(Contents[].Size)].

This feels to me suboptimal for all the usual reasons that for-loops are suboptimal. For one thing, I'd like it to be automatically parallelized on the AWS side.

It feels to me like there must be a JMESPath way to, essentially, return a list of (Prefix, Sum(Size)) pairs over all top-level prefixes. But I can't figure out how to make it happen. Adding --delimiter '/' returns only the CommonPrefixes; it doesn't return any other information (including size), so I couldn't use that to get the sum I'm looking for.

Is there some trick I'm missing? Or is the for-loop the best option I've got?

Steve Shulman-Laniel

Posted 2019-01-29T22:25:04.577

Reputation: 21

No answers