Self-hosted, cross-platform repository for large files

I'm looking for something well suited for syncing folders full of large media (eg photos, music) across several machines, ideally with revisions.

I'm using SparkleShare for text and other small files, but Git is not well suited for managing large binaries, because it wastes time trying to compress them, and it requires, at the very least, 2X + delta space.

I would have both Linux and Mac clients; Android and/or iOS would be a bonus.

If I find nothing else, it may end up being rsync scripts with cron jobs, though I'm not terribly excited about that.

Bryan Agee

Posted 2012-04-19T17:19:47.407

Reputation: 367

Git compression can be disabled. Also; is streaming on demand out of the question?

– Eroen – 2012-04-19T17:25:52.203

Yes, and I did try that. The 2X+d is still an issue, and it's painfully slow (even on an ssd). – Bryan Agee – 2012-04-19T17:27:10.283

To the streaming question: yes, it's out. We need to be able to access/modify files off-line as well. – Bryan Agee – 2012-04-19T17:40:56.897

Is Subversion out of the question? You can access/modify files off-line, but you can't commit changes without connecting. – Kevin Vermeer – 2012-08-17T17:16:41.063

Answers

You'd have to read through the docs to see if it has any revision control, but you should check out OwnCloud. It's basically like any of the other cloud hosing providers, except that you can host it yourself. It also has Mac, Windows, and Linux clients.

Thecal714

Posted 2012-04-19T17:19:47.407

Reputation: 251

If your data is not save game or similar, which must remain in the same folder structure as you found it, you can reorganize files in folders. If there are less files in a folder, the web interface will be faster. – elbedoit – 2017-12-02T22:53:53.530

I've been told it works slow, I'll give it a try. – iElectric – 2012-08-18T18:31:59.753

I guess it would depend on your hardware and setup. I've never put it under any strain, so I couldn't tell you. – Thecal714 – 2012-08-19T03:29:45.513

I use ownCloud. The web interface is slow for some reason when you have a lot of files (like both of us do!), but the sync client has good performance from my experience. – tricasse – 2012-08-23T11:51:55.513

We'll have to see how it plays out, but this is the best option I've seen, and we're going to give it a run. – Bryan Agee – 2012-08-23T21:47:10.570

Good! I hope it works out. I've been using the Windows sync client to move saved games between my different computers for a few weeks now. – Thecal714 – 2012-08-24T01:04:00.217

Regarding Git, see https://stackoverflow.com/questions/540535/managing-large-binary-files-with-git for a discussion on the subject and also see git-annex.

git-annex does not do version handling for the annexed files, but in all other aspects it works like Git, with pros and cons. Be sure to read what git-annex is not.

I also don't see rsync as an especially bad solution, if scripted nicely. It'll do very efficient transfers. The problem will be if it's hard to define a "master" drive, so to speak. If you only clone drive A to the others, everything is trivial, but if you sometimes add data at drive 2 and want that to propagate to the other disks, it becomes tedious. And what if there are conflicts... This makes solutions such as git-annex very nice. Even if one doesn't get the revision control part of Git, there are many advantages left.

Also, in my experience revisions are seldom (never?) needed for media files and such, unless one is creating music or similar, so the git-annex trade-off is OK in my book.

UPDATE: A good place to find initial information is the Kickstarter campaign that Joey Hess launched to be able to spend much more time on the project. See git-annex assistant: Like DropBox, but with your own cloud.

The rundown of git-annex on lwn.net might also be an interesting read.

Daniel Andersson

Posted 2012-04-19T17:19:47.407

Reputation: 20 465

Possibly Unison can help you out with your problem. It has Windows, Mac and Linux clients. But I don't think it does versions. Worth having a look.

trailmax

Posted 2012-04-19T17:19:47.407

Reputation: 516

You could use git-bigfiles :

git-bigfiles is our fork of Git. It has two goals:

make life bearable for people using Git on projects hosting very large files (hundreds of megabytes)

merge back as many changes as possible into upstream Git once they’re of acceptable quality

harrymc

Posted 2012-04-19T17:19:47.407

Reputation: 306 093

Its a little basic, and uses a CLI interface, and as far as I know, there's no client for the two smartphone/tablet Oses, but Boar is a version control system designed to handle binaries - it has windows clients and works both locally and over a network.

Boar's documentation also suggests rdiffbackup as an alternative to itself - runs natively on windows (though they seem to suggest you compile it yourself) and posix style OSes as well.

Journeyman Geek

Posted 2012-04-19T17:19:47.407

Reputation: 119 122