S3fs performance improvements or alternative?

Question

I've been trying to use s3fs on an EC2 server but it's really slow. I spent the last 5 hours uploading 100MB of small files.

Is there anything I can do to improve the performance or is it something related to S3fs? If not, what's an alternative I could use?

Depending on your storage needs, you may want to just use EBS, unless you need access from other systems simultaneously, in which case setting up a virtual nas with EBS may still be a better option. — Tracker1, Dec 20 '14 at 22:45

score 11 · Accepted Answer · edited Oct 15 '14 at 09:02

11

S3FS may not be the best choice for a large amount of smaller files. The overhead with S3FS is pretty high as well. I'd suggest using something like S3Curl

You can even get parallel transfers going. Just remember it will never be fast like EBS / local storage.

If you need to have it as a 'mountable' storage, the only alternatives to S3FS that I know of is S3Backer or s3ql

edited Oct 15 '14 at 09:02

ohrstrom

138
8

answered Jun 06 '12 at 14:24

thinice

4,676
20
38

1

I'm looking for something that I can mount as a filesystem. – MB. Jun 06 '12 at 14:27
Any thoughts on `s3ql`? – ewwhite Jun 06 '12 at 14:29
I'll try S3Backer and S3Sql. Thanks. – MB. Jun 06 '12 at 14:32
5 hours is really slow for ~500, 200KB files even with S3FS. Something else might be factoring in, I'm not aware of any S3FS tunable that would help – thinice Jun 06 '12 at 14:32

score 9 · Answer 2 · answered Oct 17 '15 at 00:36

9

I just released v0.0.1 of https://github.com/kahing/goofys which was partly motivated by performance problems in s3fs. File creation speedup is 3-6x and time to first byte is 58x. Feedback welcome!

answered Oct 17 '15 at 00:36

khc

306
2
2

3

As of 2019 - Goofys is the recommended choice. Riotfs hasn't been updated in a while. – Timofey Drozhzhin Sep 30 '19 at 01:38
2

2021 checking in - Goofys is incredible. According to my droplet graphs I've gone from 20% CPU to 1% CPU, load is now negligible, mem usage is flat at 20% (instead of spiking to 80%), and bandwidth is down from a sustained 1Gbps to under 500 kbps. I don't understand it, but the same workload is running and average operation times dropped from 6.5 seconds to 1.8. And the worst case is 10 vs 110. These are small random reads in very large files. – Aaron McMillin May 12 '21 at 17:39

Milo de Vries · Answer 3 · 2014-09-22T09:18:34.047

5

I just benchmarked riofs compared to s3fs. My test case was a relatively simple bash script that ran pngquant on each .png it found. On a testbucket with ~70 images of which ~20 where png (spread over a lot of subdirectories, which prob slows things down) the results where:

s3fs: 3m54
riofs: 15.9s

So for this testcase riofs is ~15x faster! Setup was also very straightforward, although the docs are somewhat terse.

Regarding the fact that the script still took 15.9s: it's not very efficient, also actually running pngquant on png's is a cpu intensive process.

edited Sep 22 '14 at 09:18

answered Sep 22 '14 at 08:57

Milo de Vries

79
1
3

Unfortunately riofs fails with ```AWS message: The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.``` – lrkwz Jan 23 '17 at 16:56

score 3 · Answer 4 · answered May 08 '21 at 01:08

https://github.com/kahing/goofys

Goofys allows you to mount an S3 bucket as a filey system.

It's a Filey System instead of a File System because goofys strives for performance first and POSIX second. Particularly things that are difficult to support on S3 or would translate into more than one round-trip would either fail (random writes) or faked (no per-file permission). Goofys does not have an on disk data cache (checkout catfs), and consistency model is close-to-open.

score 1 · Answer 5 · answered Sep 30 '12 at 21:15

1

I used [1] and it's very good. It's lot of commands and makes accessing aws and s3 a lot easier.

http://timkay.com/aws/

answered Sep 30 '12 at 21:15

Amit

111
1

1

Welcome to Server Fault! Whilst this may theoretically answer the question, [it would be preferable](http://meta.stackexchange.com/q/8259) to include the essential parts of the answer here, and provide the link for reference. – Scott Pack Oct 01 '12 at 02:27

score 1 · Answer 6 · answered Mar 27 '14 at 23:42

1

Just adding riofs to the mix. I could not see any benchmark for now but there is an issue/request that anybody can do.

answered Mar 27 '14 at 23:42

f01

406
4
8

score 1 · Answer 7 · answered Apr 25 '15 at 15:32

1

I compared s3fs with riofs, and almost switched because of noticeably better performance. Problem is riofs is still not updated to v4 signatures, which means it won't work with Frankfurt or China s3 buckets. According to the owner, an update is on the way. If you don't use Frankfurt or China, riofs is a hands down better option.

answered Apr 25 '15 at 15:32

Steve

11
1

Does not work from eu-central-1 either – lrkwz Jan 23 '17 at 16:57

score 1 · Answer 8 · answered Aug 22 '18 at 22:52

1

This is kind old but may help who gets here.

I'm using aws cli and its working very good (3 Mb/s)

pip install awscli --upgrade --user
nano ~/.bash_profile
export PATH="~/.local/bin:$PATH"
aws s3 sync /sourcedir/ s3://bucketname/destdir/

answered Aug 22 '18 at 22:52

mariomol

111
2

score 0 · Answer 9 · answered Sep 06 '22 at 08:31

0

Used rclone mount and is far much faster than s3fs

answered Sep 06 '22 at 08:31

Antony Gibbs

425
2
12

S3fs performance improvements or alternative?

9 Answers9