How to transfer data between s3 buckets of the different aws account using command line

Question

How to transfer data between s3 buckets of the different aws account using s3cmd ?

command will be like this: s3cmd rsync s3://acc1_bucket/folder/ s3://acc2_bucket/folder --recursive

But then how it'll identify second bucket environment ? What is the proper way of doing it ?

score 6 · Answer 1 · answered Mar 08 '16 at 13:43

You can use Minio client aka mc, its Open Source & compatible with AWS S3.

Installing Minio client on Linux

$ wget https://dl.minio.io/client/mc/release/linux-amd64/mc
$ chmod 755 mc
$ ./mc --help

Adding AWS S3 credentials

$ ./mc config host add mys3one https://s3.amazonaws.com BKIKJAA5BMMU2RHO6Izz V7f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12

$ ./mc config host add mys3two https://s3.amazonaws.com BKIKJAA5BMMU2RHO6IBB V7f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSaa64

Using mc mirror to transfer object/bucket from one account to another.

$ ./mc mirror mys3one/photos mys3two/photos2

In this example, s3 account with alias "mys3one" with bucketname "photos" is getting mirrored to s3 account with alias "mys3two" with bucketname photo2.

You can automate the same by adding this to cron, so periodically whenever needed content can get synced.

Hope it helps. Disclaimer: I work for Minio

This is the only answer which works across S3-compatible providers, eg. when you want to migrate your data from Amazon S3 to DreamObjects or something else. — GDR, Jun 29 '17 at 18:36
This tool works wonders on mirroring Ceph to/from AWS :). Thank you! — quizac, Jan 24 '20 at 20:32

Tom · Accepted Answer · 2015-08-20T22:24:14.860

3

you simply have to authorize your IAM user to access to the buvket, within the S3 bucket policy, like so:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "whatever",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "<ARN OF YOUR IAM USER>"
                ]
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::<YOUR BUCKET NAME>", 
                "arn:aws:s3:::<YOUR BUCKET NAME>/*"
            ]
        }
    ]
}

Then, because this is cross account, you also have to allow your IAM user to perform S3 calls, by attaching a policy to your IAM user, like so:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "whateveryoulike",
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::<BUCKET_NAME>",
                "arn:aws:s3:::<BUCKET_NAME>/*"
            ]
        }
    ]
}

simply configure your cli, and you will be able to access your bucket cross account.

edited Aug 20 '15 at 22:24

answered Aug 20 '15 at 17:15

Tom

616
8
13

this is all are for single account.. not for multiple account activity – Ashish Karpe Aug 21 '15 at 06:45
this is definitely for cross account, the S3 bucket is is one account and IAM user in another one. I did this yesterday with 3 different accounts, and works like a charm. If something is not clear, please ask and I will explain more into details – Tom Aug 21 '15 at 06:54
wich bucket I have to put those policies? in the source bucket? tks – jaspion Dec 02 '15 at 17:27
This is to grant access to a given bucket for a given IAM user. So you can apply this policy on both source and destination buckets – Tom Dec 02 '15 at 22:39
Just to clarify a couple things. In each bucket policy the is the name of that bucket. In the IAM user policy, you need entries for the buckets from each account (for 4 total resource entries). – John Eikenberry Dec 08 '17 at 21:27

score 1 · Answer 3 · answered Aug 20 '15 at 14:36

1

Accounts use different credentials. There's no way to sync from one account to another and provide separate credentials. The only way to do that is to allow public access to the second bucket with a temporary account.

If that is not an option, then your best bet is to:

s3 sync s3://acct1/bucket . --profile acct1 s3 sync . s3://acct2/bucket --profile acct2

answered Aug 20 '15 at 14:36

Marc Young

141
5

that s not true, you can configure buckets for cross account access. However, your workaround is maybe sometime simpler than the cross whole account setup – Tom Aug 20 '15 at 17:16

How to transfer data between s3 buckets of the different aws account using command line

3 Answers3