VM ram requirement for aws s3 sync job

Question

I have a VM setup to sync 15TB of data split in 25GB volumes to S3 using the latest awscli v.2. I have a 100Mbps upload capacity, and the job will probably take 16 days. I haven't had any experience with this until now, I am am trying to limit the problems I might run to.

Is there any guess what would be the necessary RAM requirement of the VM to go through this process?
Is a 2 core VM OK?
Is there something else I might need to check before proceeding?

Thanks

A Raspberry Pi could do this. – Michael Hampton Aug 09 '20 at 13:17 — Michael Hampton, Aug 09 '20 at 13:17
How did it go for you? – John Rotenstein Sep 14 '20 at 11:56 — John Rotenstein, Sep 14 '20 at 11:56

score 1 · Accepted Answer · answered Aug 09 '20 at 13:20

1

2GB of memory should totally work if you are using Linux.
Yes - two core are fine
Try to compress the files splits if possible, compression (even on 2 cores) could help you to speed up the process.

I have done this work with a 2-core 2GB machine without any problems. I compressed the files on a 12-core machine with gzip and saved another 30%.

answered Aug 09 '20 at 13:20

bMalum

144
1
5

there are duplicity volumes already encrypted and compressed. Thank you – ECII Aug 09 '20 at 13:40

VM ram requirement for aws s3 sync job

1 Answers1