7

Anyone have any experience using MooseFS? I want an easy distributed storage platform to store static data archive of about 10 TB and serve it to 20-40 nodes. Also I want to be able to add storage as the archive grows without having to rebuild the filesystem. I don't care if it's a bit slow. I just want it to be simple and stable. Basically from what I can see for OS X it's between MooseFS and Gluster. Any other suggestions?

brown.2179
  • 171
  • 1
  • 4
  • I would image some people do have experience with MooseFS. Do you have an actual question? – John Gardeniers Jan 21 '11 at 00:03
  • I would like to know if any other folks have used MooseFS and if they have had a good experience with it. If not, what other distributed fs/storage system would they recommend. Pretty straight forward John. – brown.2179 Jan 21 '11 at 00:33
  • Since MooseFS became proprietary I recommend to use its GPL-3 licensed fork -- [LizardFS](http://lizardfs.org). I have exceptionally good experience with LizardFS. – Onlyjob Dec 28 '14 at 04:26

5 Answers5

3

I've played around with both systems, not production though. Gluster is nice, installation incredible easy, but MooseFS felt much more industrial and robust, and web monitoring definitely helps. I'd give MooseFS a try. Just remember to install a metalogger for backup.

Fernando G. Testa
  • 131
  • 1
3

My experiance with MooseFS is rock stable. I use Ucarp to HA the MooseFS metadataserver with meta data loggers. Set Moosefs replication goal to 2-3 in MooseFS. Use MooseFS CGI to monitor the cluster replication state. Experience are MooseFS writes are a little bit slow, reads are super fast.

My experience from before of GlusterFS is not stable. I would provoke it with test cases things like a three node cluster. For example I tried a test with a sample Linux iso file, shutdown a node, write 0 bytes to the sample iso file. Bring up the shutdown node again. See file size glitch on the clients between the shutdown node file size and the live nodes file size. It just short flipped size, before it resumed the final file size. Depending on your use cases this may or may not be acceptable.

I would either use MooseFS or Ceph with XFS more leaning to MooseFS since its been in over seven years in production use by its authors.

//Christian

Christian
  • 317
  • 1
  • 2
  • 8
  • 2
    A word of warning about Ceph -- unlike MooseFS/[LizardFS](http://lizardfs.org), Ceph do not care about data integrity. Ceph looks very impressive on paper but in reality it is very fragile and not trustworthy. After extensive evaluation of Ceph and LizardFS I recommend only LizardFS. IMHO Ceph is not suitable for any serious use. – Onlyjob Dec 28 '14 at 04:31
  • Opposite to what @Onlyjob said ceph is rock stable, fast, handles all kinds of failure well and does integrity detection and checks *and did in 2014 as well). It is an easy to install yet pretty complex system with lots of knobs and switches for a knowledgeable admin - MooseFS may be a simpler solution for smaller clusters. – grin Sep 23 '20 at 08:59
2

I just installed moosefs on 8 servers(8 core, 6 2tb drives), and it worked very well. Its a little slow(50-70MBs). one server died and moosefs rebalanced the data across the remaining systems without loosing any data.

tomas316
  • 21
  • 2
2

I've only had good experiences with MooseFS. After testing Ceph on a 6-node cluster for a few weeks and experiencing data loss, I switched over to MooseFS and it worked great, both in terms of speed and stability, and still does. Having a system that is fully distributed without a master node could be nice, though.

Alexander
  • 121
  • 3
  • 1
    I had similar experience except that I used Ceph a lot longer and had more incidents of data loss. Switching from Ceph to [LizardFS](http://lizardfs.org) (fork of proprietary MooseFS) was a big relief and a very positive experience indeed. – Onlyjob Dec 28 '14 at 04:35
1

I haven't seen MooseFS in a production environment, but GlusterFS is a mature product that I've seen work well in large and small installs.

If you haven't read the MooseFS Wikipedia article you may want to check it out:

http://en.wikipedia.org/wiki/Moose_File_System

Highlights:

  • No clustering for the metadata server: SINGLE POINT OF FAILURE!
  • Striping ... for better or worse.

OpenStack object based storage is a newcomer brought to you by Rackspace and NASA. It might be worth looking into.

rcoup
  • 167
  • 1
  • 8
Antonius Bloch
  • 4,480
  • 6
  • 28
  • 41
  • Indeed, I did read the wiki about the single point of failure. I was on the MooseFS website and according to them swapping metadata server in the event of failure "should take less than a minute". I think I'll go the Gluster route for now b/c setup looks easy on Mac OS X. I have some linux boxes which i'll do a POC of OpenStack on. Thanks! – brown.2179 Jan 28 '11 at 15:04
  • 1
    Network block storage (glusterfs) and network object storage (openstack object storage) are very different beasts. – yfeldblum Jun 22 '11 at 16:00