2

I must execute a rsync ( not only copy or move ) from all my buckets from s3 to google cloud. I need create the same infraestructure of s3 to google. I have like 2Tera in s3, and all days I receive new files..

My solution today is

gsutil rsync -r -m s3://bucket gs://bucket

But this solution is slower .

Do you have an ideia to faster?

Thanks.

Bruno Rossi
  • 21
  • 1
  • 2

2 Answers2

1

If you are synchronizing a large amount of data between clouds you might consider setting up a Google Compute Engine account and running gsutil there. Since cross-provider gsutil data transfers flow through the machine where gsutil is running, doing this can make your transfer run significantly faster than running gsutil on your local workstation.

Reference: https://cloud.google.com/storage/docs/gsutil/commands/rsync

Kamran
  • 1,415
  • 7
  • 16
1

The up to date answer to this question is to use Google Transfer Service:

What is Storage Transfer Service?

Storage Transfer Service transfers data from an online data source to a data sink. Your data source can be an Amazon Simple Storage Service (Amazon S3) bucket, an HTTP/HTTPS location, or a Cloud Storage bucket. Your data sink (the destination) is always a Cloud Storage bucket.

Motin
  • 111
  • 3