4

I need to know what the best (read: highly performant and fault tolerant) filesystem solution based on Amazon's S3 is. I have looked a s3fs, but I notice that it has not been worked on for a while now, and that got me thinking about stability and features.

What other filesystem solutions have you used and what has your experience been with it?

Andrew Gaul
  • 225
  • 1
  • 4
QWade
  • 877
  • 1
  • 11
  • 17

4 Answers4

5

Maybe S3QL will be interesting to you. I am not going to praise it here (because I'm the developer), but there is a comparison of S3QL to other S3 file systems at http://code.google.com/p/s3ql/wiki/other_s3_filesystems.

Nikratio
  • 635
  • 5
  • 13
3

You're not going to get anything that's "highly performant" (at least when compared to modern disk storage systems) when dealing with the latencies inherent with using remote storage. Performance is further hampered by the fact that the target "device" isn't a block device, which necessitates an abstraction layer that even further inhibits performance. S3 is at its best when used for what it was intended for, storing and retrieving individual files using http(s).

Regarding fault tolerance: here's an area where S3 excels. Standard S3 buckets will survive the failure of two of Amazon's facilities, which is quite good for what you're paying. If your needs aren't that stringent, they also recently released the slightly less redundant (but ~30% cheaper) RRS buckets, which are configured to survive the failure of one of Amazon's facilities.

EEAA
  • 108,414
  • 18
  • 172
  • 242
  • Erik, thank you for sharing your opinion and thoughts. I should have qualified my "performant" requirement with the caveat that I understand a remote, abstracted "filesystem" will not meet the performance of a local FS. That being said, with the multitude of s3f2 solutions out there, have you used one, and if so, why did you choose that one over another. Bottom line, I will be implementing one, and I would like to know before I do, which one has the best track record. – QWade Sep 26 '10 at 14:01
  • Understood. Yes, I've used this s3fs version: http://code.google.com/p/s3fs/wiki/FuseOverAmazon . It worked fairly well. One nice thing is that it supports disk cache, which can help speed up IO a fair amount. – EEAA Sep 26 '10 at 17:21
0

Nasuni is a startup with a relatively low-cost cloud storage appliance. Worth looking into if you need to protect TBs of data on S3 simply.

p.s. I am not affiliated with the company, just a fan of the technology.

MikeH
  • 1
-1

You can also try Amazon EFS, ObjectiveFS or SoftNAS.

Mark Jenson
  • 11