11

I have a basic cloud running on Ubuntu Server (9.04) and Eucalyptus. Walrus (Eucalyptus' API compatable S3 implementation) stores files by on the cloud controller. However each of the other 4 server have 1TB storage which is largely unused. I am looking for a way to pool all the storage together in order to make use of all available resources. I have been loooking at various options including PVFS, Lustre, HDFS (Hadoop).

My only requirments are that it need be scalable and that it runs well on Ubuntu. I would appreciate hearing from anyone who has experience with such technologies and I look forward to hearing your suggestions.

Jaunty
  • 151
  • 1
  • 1
  • 6

11 Answers11

5

While I haven't personally implemented it anywhere in our systems, I have looked pretty extensively at Gluster. I know a few people at some large sites that use this and it apparently works really well. They use it in production for some heavy duty HPC applications.

Kamil Kisiel
  • 11,946
  • 7
  • 46
  • 68
2

GlusterFS would seem like the ideal solution to me. To the guy who claims that Gluster takes lots of effort to set up I've got to say that he's probably never tried. As of Gluster 3.2 the configuration utilities are pretty awesome and it takes 2 or 3 commands to get a gluster volume up and sharing on the network. Mounting gluster volumes is equally simple.

On the plus side it also gives you a lot more flexibility than NFS. It does striping, relication, georeplication, is of course POSIX compliant and so on. There is an extension called HekaFS, which also adds SSL and more advanced Authentification mechanisms, which is probably interesting for cloud computing. Also it scales! It is F/OSS and is being developed by RedHat who've recently purchased Gluster.

juwi
  • 573
  • 5
  • 14
1

Putting some sort of shared filesystem behind a virtualization environment is pretty common. You have lots of choices, depending on what you're looking to accomplish.

The simplest solution is probably NFS, because this is going to be supported natively by whatever distribution you're running. NFS can perform reasonably well as a virtualzation backend filesystem, although it's not going to be the fastest thing out there.

If you're running a RedHat (or derivative) cluster, you'll have good out-of-the-box support for GFS2, RedHat's cluster filesystem. This doesn't scale up to hundreds of nodes, but it's fine for smaller clusters.

Beyond that, you're starting to enter the range of things like Lustre, Glusterfs, GPFS, and so forth. These are all high-performance parallel filesystems, but they require substantially more work to set up than the other options here. If you have a large environment they may be worth looking at.

larsks
  • 41,276
  • 13
  • 117
  • 170
1

i'd agree with @larsks in that NFS is the best option; set up some iSCSI targets, NFS, done. this will scale to about 5-10 nodes; YMMV based on I/O, network capability, etc. (alternatively, set up iSCSI with multipath I/O support).

If you need something about 20+ nodes, you may want to investigate Ceph. Lustre is promising and stable, but is an (F/OSS) Oracle product and I have personal dislikes against Oracle. :)

Ceph is also quite active; the most recent release was 5 days ago.

brent saner
  • 430
  • 2
  • 7
1

Have you ever looked at mogileFS? http://danga.com/mogilefs/

It's not a file system in the traditional sense, but it is good for distributing file data across a cluster (with replication and redundancy taken into account).

If you're serving up files for a web application you will need something to serve the files. I would suggest a PHP script that uses the HTTP request as the search key for finding the file you want in the mogile FS. You can then read the contents of the file into a buffer and echo/print it out.

MogileFS is already pretty quick, but you can combine mogileFS with memcache to speed up access to the most commonly used files.

Shaun Kruger
  • 111
  • 3
  • I seem to recall that MogileFS had a single point of failure, in the case of the metadata node. HDFS has a similar problem. – David Pashley Jul 05 '09 at 15:53
  • MogileFS seems quite interesting and it's replication and redundancy is ideal however it would appear to be unsuited for general usage as the applications need be aware that they are running on it. A more traditional filesystem whereby applcations can be FS agnostic would be better suited. – Jaunty Jul 05 '09 at 16:37
  • 1
    MogileFS can have several trackers running, and you can have failover mysql at the backend. This way you can remove all the single points of failure. – davidsheldon Nov 03 '09 at 23:31
1

With Lustre you have to have a special kernel on the servers, and I would only have the servers being servers and nothing else.

Strangely the most sane answer much well be NFS. We have used NFS on Amazon's cloud. It may not scale as well as some file systems but the simplicity should not me overlooked. A single name space is probably not worth the effort it would take to implement.

James
  • 2,212
  • 1
  • 13
  • 19
1

XtreemFS could be a solution for you. It is fairly simple to install and configure, there are also packages for Ubuntu.

blong
  • 144
  • 1
  • 1
  • 13
1

Are you still looking into HDFS? One of the Cloudera guys gave a talk at VelocityConf this year about Hadoop and HDFS focused on managing big data clusters, so he talked about HDFS quite a bit. The slides are pretty informative. I haven't worked with HDFS personally, but I talked with some random folks at Velocity that are using it on Ubuntu to do various data analysis.

jtimberman
  • 7,511
  • 2
  • 33
  • 42
1

MooseFS (Distributed File System) fits to your requirements. It is scalable and works well on Ubuntu. It might also be useful for you to see how to install install/update MooseFS from officially supported repository on Ubuntu.

TechGeek
  • 161
  • 2
0

Not sure what you're doing, but this sounds like a potentially interesting application for CouchDB.

duffbeer703
  • 20,077
  • 4
  • 30
  • 39
0

You could try PVFS2. It's much easier to set up than Lustre, and generally faster than Gluster.

wazoox
  • 6,782
  • 4
  • 30
  • 62
  • faster than Gluster? can you share any data around performance? – JMS77 Jan 14 '11 at 17:20
  • I have no recent benchmarks at hand. Back in 2008, Gluster was much slower than Lustre, while PVFS2 was only slightly slower. It may be different nowadays, maybe I'll benchmark this. – wazoox Jan 15 '11 at 17:34
  • thanks, if you do benchmark, please let me know. maruti.j@gmail.com or you may make a post on serverfault for everyones benefit – JMS77 Jan 19 '11 at 18:07