Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [s3cmd]

s3cmd is a simple command-line interface to storing and retrieving data to/from Amazon's S3 cloud storage service.

43 questions
8
votes
5 answers

how to pipe a mysql dump to s3cmd

I want to transfer a mysql dump, compressed, to s3. I tried: mysqldump -u root -ppassword -all-databases | gzip -9 | s3cmd put s3://bucket/sql/databases.sql.gz but then I get: ERROR: Not enough paramters for command 'put' How can I do this (in one…
zensys
  • 299
  • 3
  • 9
  • 19
7
votes
2 answers

s3cmd: Deleting files or folder does not work

s3cmd s3://MYBUCKET/FOLDER does not work. (folder is empty) s3cmd s3://MYBUCKET/FILE does not work. Output of both commands is "File FILE sucessfully deleted", yet the file remains on the bucket. Deleting with the S3 Webinterface works just fine.
user54883
5
votes
1 answer

Linux server sync to an Amazon S3 bucket

I am looking for a stable solution to replace a classic server backup to another server using rsync. I have to sync a whole filesystem (more than 1Tb) to Amazon S3. Where am I? Solution 1: I mapped the S3 bucket to a mounting point in the system…
deadtired
  • 141
  • 1
  • 7
4
votes
0 answers

S3 restoration using s3api get-object is not working in aws china region

I have set up a daily backup script in my aws china instance which uploads my required files to be backed-up to s3 bucket. I have a restoration script which uses s3api to restore the objects to the instances. The command used to get the file is aws…
Arjun Prasad
  • 163
  • 1
  • 11
4
votes
2 answers

Amazon S3 - private file is still downloadable for everyone?

I have some secret files which I want to restrict anonymous user from viewing/downloading. I have tried to make it private by running something like: s3cmd setacl --acl-private s3://bucket/some/path/*.ext Then I go to S3 Management Console, select…
quanta
  • 50,327
  • 19
  • 152
  • 213
4
votes
1 answer

s3cmd with --delete-removed

I'm currently writing a script to sync files in s3 buckets with s3cmd. I check the document and it says: s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR also I find a nice option: --delete-removed Delete remote…
lazyka
  • 43
  • 1
  • 3
4
votes
1 answer

s3cmd put only files that do not exist in destination bucket

I've been using s3cmd to sync local files on our server with the S3 bucket. There are over 200,000 files on the local server. Can't seem to find any documentation on the site that explains how to upload ONLY files that do not exist in the…
FurryWombat
  • 155
  • 1
  • 4
3
votes
0 answers

Solution for file data backup, versioning and replication

We have a growing set of data files (.wav files, image files etc) which are data, i.e not part of the application code - uploaded and modified by users. The number of files is in the 1000s and the total size reaches GBs. We have several server…
Amir Abiri
  • 227
  • 1
  • 3
  • 7
3
votes
1 answer

Sync files from local filesystem with S3 - one way

I have a process that generates log files to the local filesystem, and files that are more than 2 weeks old are automatically purged. I want to use the aws command line tools to sync my folder with S3, with the following behavior: No sync of files…
Scott Switzer
  • 133
  • 1
  • 3
2
votes
2 answers

How to upload a large file using aws commandline when connection may be unreliable?

I've been having trouble uploading a large (800MB) file to s3, using the aws commandline tool. The first attempt completed (after many hours) but was not visible, and I was advised (here) that it has been eaten by goblins and I need to start…
Darren Cook
  • 730
  • 2
  • 9
  • 16
2
votes
3 answers

How to transfer data between s3 buckets of the different aws account using command line

How to transfer data between s3 buckets of the different aws account using s3cmd ? command will be like this: s3cmd rsync s3://acc1_bucket/folder/ s3://acc2_bucket/folder --recursive But then how it'll identify second bucket environment ? What is…
Ashish Karpe
  • 277
  • 2
  • 5
  • 19
2
votes
2 answers

Logrotate script not uploading to s3

I am trying to use logrotate to upload log files from my rails and nginx commands once a day. When I run the logrotate command manually the files successfully upload, but it never does it on its own. I have tried using lastaction and also postrotate…
John
  • 123
  • 4
2
votes
2 answers

How best way to rsync from aws s3 to gcloud?

I must execute a rsync ( not only copy or move ) from all my buckets from s3 to google cloud. I need create the same infraestructure of s3 to google. I have like 2Tera in s3, and all days I receive new files.. My solution today is gsutil rsync -r…
Bruno Rossi
  • 21
  • 1
  • 2
2
votes
1 answer

mail: Cannot parse address

This is the command I use to tar my backup files, write to stdout and then upload everything to Amazon S3 with an email notification. tar -czvf - --exclude-caches /var/www | s3cmd --reduced-redundancy --multipart-chunk-size-mb=30 put -…
MultiformeIngegno
  • 1,627
  • 9
  • 24
  • 31
2
votes
1 answer

s3cmd: on-the-fly backup

To do backups I created a script that makes an archive of all the folders I need to backup, sends it to S3 (through s3cmd) and then deletes it after upload has been completed. I'm looking for a way to avoid having to create the archive and then…
MultiformeIngegno
  • 1,627
  • 9
  • 24
  • 31
1
2 3