0

Does anyone have experience with using Openstack Swift object storage with s3fs and having multiple write mounts accessing the same file(s) at the same time?

The man page for s3, for example: https://linux.die.net/man/1/s3fs has this paragraph:

"Multi User capability While it is possible to share s3 buckets among multiple users, the current data consistency model for Amazons S3 service prevent the safe usage of multiple mounts from multiple users. While s3fs will currently allow multiple mounts, data corruption may result from such activity. A future release will contain a locking mechanism to safely guard against multiple read-write mounts. multiple read-only mounts following a single read-write mount is safe, but will not reflect changes made by the writeable mount to any of the file or filesystem metadata, limiting its usefulness there"

Is the locking mechanisim mentioned above implemented yet?

Thanks Mark

mlum
  • 1

1 Answers1

0

This man page refers to a different Python-based s3fs which fell off the Internet around 2016 and ceased development in 2009:

https://web.archive.org/web/20161008081920/https://fedorahosted.org/s3fs/

I recommend using the more popular s3fs-fuse instead. I emailed the die.net webmaster to try to update this since people usually refer to the s3fs-fuse.

Specifically s3fs does not have POSIX semantics in the presence of multiple writers which may manifest as data corruption. The underlying S3 object storage follows a last-writer-wins model where objects are replaced atomically by each PutObject or CompleteMultipartUpload RPC. s3fs also has a caching model which can provide stale data to the application.

yas3fs is similar to s3fs-fuse and implements locking via Amazon SNS/SQS. However this takes you further outside the OpenStack ecosystem.

Andrew Gaul
  • 225
  • 1
  • 4