7

I need to build a solution to host internal git repositories. It needs to supports hundreds of thousands (or more) repositories.

I plan on using multiple "dumb" servers with a shared storage, so basically when a client is trying to access a repository - it will be redirected by the load-balancer to any of the available servers. Any change to the repository - will be replicated across all nodes.

My first thought was to use GlusterFS for that, but I've read it doesn't handle well with small files. I'm also thinking of replicating everything myself using DRBD, but this requires more setup and seems more complicated when comparing to GlusterFS.

Which one of the two provides better performances? Basically the problem I'm trying to solve is that when any of the servers goes down - I want others to still be able to serve the data.

Gilad Novik
  • 307
  • 2
  • 3
  • 10

5 Answers5

5

This is a classic scale-out use case, and IMO GlusterFS should fit the bill. You can give it a try - just bring a few VMs up, set up a few bricks to be used for repository storage and run a stress test.

DRBD is not an option here anyway - it doesn't scale. If anything, I'd look at other object storage projects (Swift for example), if Gluster doesn't work well enough, but none of them are extremely performance oriented

dyasny
  • 18,482
  • 6
  • 48
  • 63
  • 1
    It seems like GitHub are using DRBD for their storage (I guess it does scale), but it seemed like an overkill IMO, and I do prefer a distributed file-system solution where everything "just works". – Gilad Novik Jan 28 '14 at 16:49
  • Well, the fact that someone uses it, doesn't mean it scales. It only means they have some failover clusters using DRBD, which is not the same as multiple copy active/active file access nodes you can get with gluster/swift/ceph/etc – dyasny Jan 28 '14 at 17:23
  • Does DFS's performance drop with the an increasing number of nodes? – CMCDragonkai Jun 20 '14 at 05:34
  • what does DFS have to do with gluster/drbd? – dyasny Jun 22 '14 at 04:58
4

I had a similar set up for cyrus mail servers in which gluster has proven not to be able to handle the load during stress tests. we originally chose gluster because it was simple but had to move back to drbd. however as dyasny highlighted, drbd activs/active cfg is not advisable (not to mention the pain). if you need all servers to mount the shared storage at the same time drbd is not an option. Another thing you may want to look at is clvmd with lock manager. lvm2 now have support for raid type lv and use man code for that purpose. this mixed with the appropriate filesystem (some cluster aware ones if needed) can be an option. however I never tested it myself in production, only as PoC.

alxgomz
  • 1,600
  • 1
  • 10
  • 14
  • 1
    Just one thing I'd like to add, cluster didn't cope in my setup, but keep in mind it was a 2 nodes cluster. Gluster would have behaved better with more nodes. – alxgomz Jan 30 '14 at 07:35
3

The problem you will have with Gluster is mainly latency between the nodes. I would advise you give it a try and see if it handles your workload. If your servers are are powerful enough and have a fast interconnect you should get a fairly good performance. Also you might want to try the built-in NFS server, which, talking from my experience, handles small files a little bit faster.

But if you need your solution as fast as possible GlusterFS probably isn't for you. Other options would be commercial products like Git Clustering(haven't tested that, though) or you could build your own solution with free tools like gitmirror.

Izzy
  • 786
  • 2
  • 8
  • 29
  • Actually performances is not my #1 priority (few more seconds are still ok), but stability and replication is. Which NFS server were you referring to? – Gilad Novik Jan 28 '14 at 16:46
  • GlusterFS has a [built-in NFS server](http://gluster.org/community/documentation/index.php/Gluster_3.1:_Manually_Mounting_Volumes_Using_NFS). Performance is better but you have to handle the failover on the clients yourself. On the other hand you save yourself the need of installing the FUSE-client for access to the GlusterFS servers. – Izzy Jan 28 '14 at 21:13
2

If you are considering Gluster (which in my experience is usable with many small files but YMMV), you might want to have a look at Ceph as well http://ceph.com/

thanasisk
  • 941
  • 6
  • 16
  • My problem with Ceph is that it has a meta-data server, which seems to actually hurt performances since every data access will also require accessing the meta-data server. Am I wrong? – Gilad Novik Jan 28 '14 at 16:42
  • metadata servers can be duplicated too, and can run on the same nodes as object store servers. You'll need to fetch metadata at any point, whether you fetch it from the object store or from the metadata server, so it *shouldn't* significantly impact performance. I haven't actually tried any of this though. – Wouter Verhelst Nov 26 '17 at 09:08
0

I would also recomend Glustet ad the easyest, and fastest to setup , configure. Also it scales out very very well. The problem is speed, and my 2c ia that you need to choose between easy configuration and easy scaleout, and between some technical dificulties (some fancy drbd / ocfs2 / Glances block storage) with the gain of speed. But..how much speed you gain? Yoi need to make some stress test and choose..

x86fantini
  • 302
  • 1
  • 3
  • 9
  • Assuming I want to extend the storage, can I simply add another VM to the list and it will replicate to it, or will it completely block access to entire storage until this VM is ready to be used in the cluster? Is rebalancing happen asynchronously? – Gilad Novik Jan 28 '14 at 16:48
  • Gluster is brickable, so any new brick will be added to the one, single, uniqe brick.. – x86fantini Jan 28 '14 at 16:55
  • So I guess if I want the fastest way to add a brick without stopping the cluster for too long, I should only use fixed layout and not migrate data? – Gilad Novik Jan 28 '14 at 17:40