0

This used to work just fine but in the last two weeks it has been noticed that the connection seems to have slowed down significantly. Performing gsutil ls -lr gs://bucket/folder takes around 10 minutes. There are about 4000 files in that folder totaling about 250GiB. Performing the same command in Cloud Shell takes seconds.

The gcsfuse is mounted through fstab with: gcsbucketname /localfolder/gcsbucketname gcsfuse rw,noauto,allow_other,nosuid,nodev,relatime,user_id=11111,group_id=111,default_permissions

user_id and group_id changed to all 1's.

Matthew
  • 1

2 Answers2

1

You are likely hitting the ops limit in this case which is set very low by default:

If you would like to rate limit traffic to/from GCS in order to set limits on your GCS spending on behalf of gcsfuse, you can do so:

The flag --limit-ops-per-sec controls the rate at which gcsfuse will send requests to GCS. The flag --limit-bytes-per-sec controls the egress bandwidth from gcsfuse to GCS. All rate limiting is approximate, and is performed over an 8-hour window. By default, requests are limited to 5 per second. There is no limit applied to bandwidth by default.

https://github.com/GoogleCloudPlatform/gcsfuse/blob/master/README.md#rate-limiting

Consider setting --limit-ops-per-sec to a higher number, or 0 if unlimited is fine.

In /etc/fstab you use limit_ops_per_sec=0 instead.

Maeve Kennedy
  • 111
  • 2
0

Passing “--implicit-dirs” may be helpful in order to reduce ls latency according to this github issue

David
  • 101
  • 2