2
I have an S3 bucket with a bunch of top-level prefixes. I have a relatively inefficient way of summing the bytes across top-level prefixes:
- Get the list of prefixes via
aws s3 ls [bucket name]
, followed by some sed/grep. - do a bash for-loop over those prefixes, running
aws s3api list-objects --bucket [bucket name] --prefix $prefix --output json --query [sum(Contents[].Size)]
.
This feels to me suboptimal for all the usual reasons that for-loops are suboptimal. For one thing, I'd like it to be automatically parallelized on the AWS side.
It feels to me like there must be a JMESPath way to, essentially, return a list of (Prefix, Sum(Size)) pairs over all top-level prefixes. But I can't figure out how to make it happen. Adding --delimiter '/'
returns only the CommonPrefixes; it doesn't return any other information (including size), so I couldn't use that to get the sum I'm looking for.
Is there some trick I'm missing? Or is the for-loop the best option I've got?