Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [gsutil]

33 questions
28
votes
2 answers

Using JSON keys with google cloud gsutil

I have a ssh private key in a key.json file, and I want to use this credential to access a storage bucket using gsutil. I can't seem to find anything about how to include json keys as a authentication method, only "private" and "secret" fields. The…
GuySoft
  • 425
  • 1
  • 4
  • 11
6
votes
1 answer

Slow synchronisation stage on gsutil rsync?

I've just started to use GCS as backup for my web servers. One server has 1.2 million JPEGS (3.5TB) and this all rsynced over flawlessly over 10 hours or so. The other has 2.5 million JPEGS (just thumbnails/previews though - 300GB total). The…
Codemonkey
  • 1,034
  • 2
  • 17
  • 36
3
votes
1 answer

gsutil rsync "too many values to unpack" error

I'm trying to sync a folder with over 3k files to a bucket on Google Cloud with gsutil. The problem is that less than half of the files are uploaded and then I get this error: too many values to unpack CommandException: 1 files/objects could not…
Enrico Dias
  • 133
  • 5
3
votes
1 answer

How to use JSON keys with google cloud gsutil to manage multiple Keys

We have multiple GCP Service account keys from different environment such as DEV,STAGING,..PROD. I would like to run a command in my jenkins box which is there in Dev environment to create access DEV and Staging environments, I don't want to run…
Dinesh SC
  • 33
  • 1
  • 6
3
votes
1 answer

Copy list of files (-I flag) with gsutil preserving path

I am trying to copy all pictures and static files to a bucket of mine in Google Cloud Platform. I am attempting this command from the root dir of my app: find -regextype posix-extended -iregex ".*\.(js|css|png|jpg|gif|ttf|cur|woff|eot)" | gsutil -m…
santiago arizti
  • 405
  • 5
  • 16
3
votes
1 answer

ServiceException: 401 Unauthorized WebHook callback channel

Can anyone help me with issue on setting up "Object Change Notification". I am trying to execute " gsutil notification watchbucket https://.appspot.com/ gs://glrs_test_deepshikha It is giving error : ServiceException: 401…
Deepshikha
  • 31
  • 2
2
votes
0 answers

gsutil rsync fails on Ubuntu with UnicodeDecodeError

I'm trying to rsync between google cloud storage and amazon s3. On my local machine (MacOS) this command works gsutil rsync -d -r gs://mira-internal/data/exports/20191108 s3://mira-temp/raw/20191108 However, when I try to run it on a Ubuntu machine…
kellanburket
  • 71
  • 4
2
votes
1 answer

Google Storage ACL - write only for public, but don't allow read, list, delete or update operations

I need to setup Google Storage access for 3rd parties (public) to allow them to upload one or more files, but they shouldn't be allowed to read, list, delete or update an existing file. This structure allows anyone to upload the data to that bucket,…
Prashant
  • 21
  • 1
2
votes
1 answer

gsutil ls -l fails when gsutil mb succeeded

I get the following when I attempt to list the buckets for the account I'm logged in with: $ gsutil ls -l gs://bucket AccessDeniedException: 403 Forbidden $ gcloud config list Your active configuration is: [default] [compute] region =…
Cognitiaclaeves
  • 181
  • 10
2
votes
1 answer

Setting up logging for buckets in Google Cloud storage

Trying to set up logging on a GCS bucket. every time i enter the command to set logging i get the error below. "Command exception: "logging" command does not support "file://" URLs. did you mean to use a gs:// URL?" is it my test logs bucket that…
Tony Ugarek
  • 21
  • 2
2
votes
1 answer

gsutil cp error: cannot have start index greater than total size

I'm trying to download a lot of images from a Google Bucket that my employer's web host (not so) helpfully set up. I run the command gsutil cp -r gs://bucket-link-info/images/ . and the download starts fine. However, after a while, it stops with the…
GK Masterson
  • 21
  • 1
1
vote
1 answer

How/why would ALL our Google Cloud Platform Service account keys suddenly disappear/be deleted?

We had a major issue with Google Cloud Platform some time between 2019-04-16T11:00:00Z and 2019-04-17T02:30:00Z, when all of our service accounts lost their access keys for an unknown reason, so our Google Compute Engine and Google Cloud Storage API…
Sim Kennedy
  • 31
  • 3
1
vote
1 answer

CommandException: arg [...] does not name a directory, bucket, or bucket subdir. for a directory on Compute Engine

I am trying to use gsutils command to deploy changes from CircleCI to a directory on my Compute Engine instance. The command I am using is: gsutil -m rsync -d -r dev/ [VM_INSTANCE_NAME]:/var/www/dev/ but I get the following error: CommandException:…
Jorjani
  • 123
  • 3
  • 10
1
vote
1 answer

Access denied 403 when trying to download from the Google Cloud Platform, but user has all access

We are just trying to use gsutil from the command line to download some DCM data from the Google Cloud Platform: gsutil -m cp -R gs://dcdt_-dcm_account75701/dcm_account75701_activity_201803* C:\Users\[omissis]\Desktop\ImprData We are getting the…
Matt
  • 11
  • 1
  • 2
1
vote
2 answers

Convert filenames to lowercase [Google Cloud Storage] [gsutil]

We recently moved all product images from our website to Google Cloud Storage. Due to bad naming policies in the past the casing (lowercase, uppercase) in the file names does not always match the way file names are listed in our product database.…
m4v21
  • 21
  • 3
1
2 3