Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [s3fs]

s3fs is FUSE-based filesystem that stores files in AWS S3.

s3fs is FUSE-based filesystem that stores files/data in AWS S3.

64 questions
17
votes
9 answers

S3fs performance improvements or alternative?

I've been trying to use s3fs on an EC2 server but it's really slow. I spent the last 5 hours uploading 100MB of small files. Is there anything I can do to improve the performance or is it something related to S3fs? If not, what's an alternative I…
MB.
  • 375
  • 2
  • 5
  • 12
15
votes
5 answers

How to make s3fs work with IAM roles?

We are using IAM roles to maintain the machines. We now planning to mount the s3 to our ec2 instances and do the processing as per our need. We been using s3fs tool for mounting. But that doesn't seems to be working with the IAM roles. Are there any…
sriram
  • 277
  • 1
  • 3
  • 9
10
votes
2 answers

How to load balance sftp instances on AWS

I like to know is it possible to load balance sftp servers in AWS. I have 2 servers, and each of my servers are using s3fs-fuse to mount the same S3 bucket onto a mount point. Both of my ec2 instances are able to read/write to their mount points,…
popopanda
  • 201
  • 3
  • 4
9
votes
2 answers

EC2 - Shared Storage - S3FS or EBS?

Building my web service onto EC2 right now and have a single instance behind a load balancer. I will of course cater for multiple instances. My initial idea was to run all the instances a dumb slaves, and use S3 as local storage. For this, I've…
waxical
  • 344
  • 1
  • 5
  • 12
7
votes
1 answer

How to force s3fs mount on boot

I'm using s3fs 1.33 on Ubuntu 9.10 x86. I mount it manually with this command: sudo /usr/bin/s3fs mybucket /mnt/s3/mybucket -ouse_cache=/tmp How can I force the mount on boot? I guess that I have to write something to /etc/fstab. But what? And how…
Alexander Gladysh
  • 2,343
  • 7
  • 30
  • 47
6
votes
2 answers

Can not unmount an S3 directory mounted with s3fs-fuse

I have an s3fs-fuse "device" mounted at /mnt/s3. I want to unmount it so that I can remount it with the mp_umask flag so that chroot will work properly. The command should be: fusermount -u /mnt/s3 Unfortunately the result is: fusermount: failed to…
AKWF
  • 481
  • 2
  • 6
  • 17
5
votes
0 answers

How does S3FS (or any other S3 FUSE filsystem) compare to AWS FSx for Lustre + S3

I remember trying s3fs a year back, trying to use some S3 bucket as a FUSE filesystem. I remember it being quite laggy, especially when coupled with git operations on it (an oblivious system architect had a weird requirement like this). Fast forward…
dimisjim
  • 215
  • 2
  • 10
5
votes
2 answers

Automatically mount bucket with s3fs on boot

I use an Amazon S3 bucket to deliver some of my server's content. I was able to mount it successfully, and grant Apache rights over it, but can't get it mounted properly at reboot. I updated my /etc/fstab with this line, but nothing happens when I…
Balmipour
  • 294
  • 1
  • 3
  • 10
5
votes
3 answers

Replicated shared filesystem

I'm looking into setting up a shared filesystem/file server on AWS (EC2) infrastructure that offers replication and fairly painless failover. This filesystem would host potentially millions of files that are a few megs in size. Those files would be…
Alex
  • 471
  • 7
  • 18
5
votes
1 answer

Linux server sync to an Amazon S3 bucket

I am looking for a stable solution to replace a classic server backup to another server using rsync. I have to sync a whole filesystem (more than 1Tb) to Amazon S3. Where am I? Solution 1: I mapped the S3 bucket to a mounting point in the system…
deadtired
  • 141
  • 1
  • 7
5
votes
2 answers

S3 based file system capable of requesting only part of file

I'm storing large datasets in s3, but on a given computer in my cluster, my program only needs to read a small subset of the data. I first tried s3fs, but it downlooads the entire file first, which takes a really long time. Are there any s3 backed…
UsAaR33
  • 1,036
  • 3
  • 11
  • 20
4
votes
1 answer

Configure SFTP with OpenSSH and an AWS S3 Bucket mounted via S3FS on Amazon EC2

How do I allow multiple SFTP Users with S3FS and OpenSSH? Everything works, except SFTP Users don't have permission to write to their Chrooted Home Directory: remote open("/some_file"): Permission denied Setup I've got an Amazon EC2 instance running…
T. Brian Jones
  • 887
  • 3
  • 17
  • 29
4
votes
2 answers

s3fs "could not determine how to establish security credentials"

I'm trying to use s3fs to mount an S3 bucket on to a standard AWS Amazon Linux AMI (with all the necessary dependencies installed). However when following this tutorial when I run: s3fs mybucketname -o allow_other myfolder or variations thereof, I…
Chris Reynolds
  • 141
  • 1
  • 4
4
votes
4 answers

Best remote mounted filesystem based on S3

I need to know what the best (read: highly performant and fault tolerant) filesystem solution based on Amazon's S3 is. I have looked a s3fs, but I notice that it has not been worked on for a while now, and that got me thinking about stability and…
QWade
  • 877
  • 1
  • 11
  • 17
4
votes
3 answers

s3fs Input/output error

I'm trying to set up a backup system with s3fs and the Amazon S3 service. I followed this this guide and this guide. Tailing /var/log/messages I get: Aug 28 13:37:46 server s3fs:###response=403 I already tried creating the authentication file…
shadow_of__soul
  • 376
  • 1
  • 6
  • 16
1
2 3 4 5