Huge File System?

Question

Suppose that one has 100 machines (each with a 2TB hard drive), all connected in a network.

Is there a way (in Linux) to unite this combined 200TB of disk space into one folder, that can be shared using NFS among all machines that are in the network?

Going this route, all machines will be able to read/write to the same folder, and this way, I/O will be spread evenly between them. Is this a good idea if one needs a large file system to store 100s of TB of data? (note: data will be split to many smaller files, of size ~ 500GB each).

Is there a ready-made solution (prefereable, an open source one) that achieves just that?

'I/O will be spread evenly between them' -- Keep in mind that any distributed solution will require bandwidth between the distributed servers ( e.g. host1 will need to have some knowledge of activity stemming from host2 though host 100). This bandwidth is non-trivial. — Stefan Lasiewski, Jul 12 '11 at 20:17

score 7 · Accepted Answer · answered Jul 12 '11 at 19:50

7

Two options that come to mind are GlusterFS and Hadoop HDFS.

answered Jul 12 '11 at 19:50

bosmacs

186
3

Thanks; any words of info on the difference of installation / maintainance of `GlusterFS` vs. `Hadoop`? – user76976 Jul 12 '11 at 19:59
1

I haven't used HDFS; I imagine it fits will if you're already using Hadoop for parallel number-crunching. Gluster sounds more like what you're looking for -- a distributed NAS, in effect. – bosmacs Jul 12 '11 at 20:04
Great; is it hard to install? when one wants to add another node to the picture, what is the overhead? – user76976 Jul 12 '11 at 20:10

score 4 · Answer 2 · answered Jul 12 '11 at 22:16

IBM's GPFS can do this (note: not open-source).

With GPFS you can create Network-Shared-Disks (NSDs) that are comprised of any type of block-storage (local or presented via iSCSI or FC, for example). It would be entirely possible to create a GPFS filesystem (device) that is comprised of NSDs that span each 2TB hard drive across your 100 servers.

I won't pretend to recall all the crazy marketing numbers, but GPFS is among the most popular clustered file systems for supercomputers in the "Top 500" list because it supports extremely large volumes of data and incredibly high parallel I/O. Wikipedia has some numbers.

GPFS can replicate data and metadata blocks across the filesystem. When you create your NSDs you define a "failure group", you do this so that GPFS does writes your block replicas elsewhere (that is, you don't want both copies of your block in the same failure group). You can also tier storage using their concept of "storage pools", through which you can define behavior like...files accessed in the last week live on my Fusion IO or SSD drives, but after that move the blocks to cheaper storage.

All the nodes in your cluster would have access to a device (like /dev/gpfs0) which they could mount and access as it the entire file system were local to each node. You mentioned NFS; in this model however, it is not necessary to introduce the additional protocol unless you have systems outside the 100 node cluster that are acting as consumers/clients of the data and you do not wish to make them GPFS clients/NSD servers (by loading the GPFS kernel module). However, you can trivially export GPFS file systems via NFS and even leverage Clustered-NFS (CNFS) if necessary.

I do not work for IBM, but I have played with GPFS a little bit and I liked it.

thanks; I really prefer an open source solution though – user76976 Jul 12 '11 at 23:06 — user76976, Jul 12 '11 at 23:06

score 3 · Answer 3 · answered Jul 12 '11 at 19:46

3

I have not had a chance to play with it yet so I can't give a full review, but I would say have a look at the Openstack cloud stuffs -> http://www.openstack.org/projects/storage/

answered Jul 12 '11 at 19:46

Benjamin T

31
2

score 2 · Answer 4 · answered Jul 12 '11 at 19:59

2

You probably want something like PVFS.

answered Jul 12 '11 at 19:59

ninjalj

255
3
7

score 2 · Answer 5 · answered Jul 12 '11 at 20:37

2

There is a full list on Wikipedia.

answered Jul 12 '11 at 20:37

cmcginty

1,263
15
24

1

It would be better to directly link the "Distributed parallel fault-tolerant file systems" – Karoly Horvath Jul 12 '11 at 21:05

score 1 · Answer 6 · answered Jul 14 '11 at 20:45

1

Ceph is another, though not prod ready yet. http://ceph.newdream.net/wiki/Main_Page

answered Jul 14 '11 at 20:45

Ronald Pottol

1,683
1
11
19

Huge File System?

6 Answers6