How to synchronize the home folder between multiple computers?

Question

I have three computers at home, and would like to have the /home/ folder tree synchronized between the three.
Any files/folders (except those hidden) that are modified/added/removed in one of the three computers is updated in the other two.

Is rsync enough to do this? and how?

Answer 1

Here's a list things that could potentially solve this problem, each balances the trade-offs you have to make differently so you'll have to make your own choices and try things out for yourself:

• Unison - as mentioned by others, this is run manually, but is very fast, reliable and effective. Requires both machines being synchronised to be on at the same. It has a nice user interface to allow you to deal with the almost inevitable conflicts, and tracks and propagates deletions correctly. The graphical app/package is called unison-gtk.

• OwnCloud - Cloud storage run on your own server. You'll need a machine to leave on. Requires a reasonable amount of setup. Runs a full Apache 2 webserver and an SqlLite or MySQL database on the server. Works similar to Dropbox with a desktop client, but the server is under your control. edit : OwnCloud has recently gone through some changes in how the project is run, and now has a new fully open source (ie no closed source 'enterprise' edition) under the guise of NextCloud, (see this youtube interview with the original OwnCloud developer for more details).

• SparkleShare - uses git to keep files in sync. According to the homepage: good for many smaller files, not good for lots of large files such as music or photo collection.

• Seafile - Provides a server component you can install on a local machine. Seafile uses a data model similar to git for tracking changes. Provides sync clients for desktops, tablets and smartphones. A blog post describing setup can be found at http://openswitch.org/blog/2013/07/18/installing-and-configuring-seafile-on-ubuntu-12-dot-04/

• Osync - "... bidirectional file synchronization tool written in bash and based on rsync. It works on local and / or remote directories via ssh tunnels. It's mainly targeted to be launched as cron task" (text from the website)

• PowerFolder - java based GPL v2 project. Main website pushes commercial offerings so it's not clear how to use the provided .jar file.

• Rsync - fast and effective and been around for decades, however it doesn't keep a history so you have to choose a direction to decide whether a file is new or deleted. Graphical tools are available such as gwRsync.

• Lsyncd - monitors folders/files to trigger rsync replication

• dvcs-autosync - written in python, uses git to store and share changes between machines, and XMPP to communicate changes.

• git-annex - command line tool for shunting files around, based on git. There's an illustrative walkthrough here: http://git-annex.branchable.com/walkthrough/

• Tonido - freeware. Provides a desktop app that will share files to other devices. Also provide commercial cloud offerings, and the TonidoPlug plug computer.

• BitTorrent Sync (freeware) - peer-to-peer file sync based on BitTorrent. I don't know much about this as I won't be using it due to it not being open source and not trusting it to keep my data within my LAN, feel free to edit this answer with better information / real experiences.

• SyncThing - Developed as an open source alternative to BitTorrent Sync. It currently lacks some of the advanced features of BitTorrent Sync, such as untrusted peers. It is under active development.

• Commercial hosted services such as dropbox, ubuntu one, google drive, apple iCloud are all quick cheap and convenient, however they all require trusting a company with all your data, and need a reasonably fast internet connection.

• Git / subversion - Use a source control system directly. Completely manual and can be a little complex but popular approach with some users familiar with these systems from using them as programming tools.

• CloudFS - syncronise a whole filesystem, cluster technology based

• NFS mount - basically your home lives on one machine and you access it over the network, no good for laptops you take with you. More info: http://www.linuxjournal.com/article/4880

---

Factors to consider in making your decision:

• Central server - some solutions require a machine to be on all the time (or at least when you need to synchronise) for other machines to synchronise with. This could be one of your existing machines, or a separate machine such as a NAS. Watch out for increased power bills.

• Automatic / Manual / Scheduled - The best way to avoid having to resolve conflicts where something is changed on more than machine is to have a program on every machine that watches for changes and synchronises immediately, this way you reduce the opportunity to end up with multiple versions. With manual processes you always have to remember to run the synchronisation.

• Remote access - do you want to synchronise away from your LAN (aka home), think about the security implications of this.

• Security - does your data leave your network encrypted or not, how secure is the transfer between machines. What if someone captures your data on the move and later the encryption is found to have flaws? Who controls the server that keeps your data, is the data enccrypted, can you trust any third-parties? Do you have to poke holes in your router to get remote access. How long do 'deleted' files and related meta-data stick around for on the synchronised devices and on the central server. Are you synchronising between encrypted and unencrypted storage?

• Moving large folders - the solutions I've tried all have an issue that when you move / rename a file or folder the sync doesn't understand this and uploads it all over again as new and then deletes the old copy. Please help out by marking any solutions above that are capable of dealing with this (I suspect the git based solutions do as git doesn't suffer from this due to the content-based addressing it uses but don't know for sure as I haven't used them).

• Disk capacity

• Backups - synchronisation is not backup. Delete an important file by mistake and many of the above will merrily delete all your other copies. I recommend reading Mat Honan's piece on being hacked for a good account of what can happen if you put all your digital eggs in one digital basket, so to speak.

---

I recommend not syncing the entire home folder, but instead picking specific folders to sync such as Documents/, Pictures/ etc. This will avoid the pain of being forced to deal with the speed / performance / disk space issues of automatically synchronising everything. It also avoids having to maintain exclusion lists.

As I continue to try and find something that works for me personally I'll try and keep this answer up to date with useful info. I've aggregated the information from all the other answers into one complete answer.

References:

• LinuxFormat - February 2014 LXF180 p31, "Hosted Storage Roundup"

hacking/all/

Answer 2

Unison might be a good candidate:

Unison is a file-synchronization tool for Unix and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other.

It already does 2-way syncs. See update bellow.

I've learnt that there are very few things rsync cannot do, and it can probably provide an equal or better solution, but you'll have to wait for an rsync expert to turn up for that solution.

Update: Yes, Unison can sync more than 2 machines. From their user manual:

Using Unison to Synchronize More Than Two Machines

Unison is designed for synchronizing pairs of replicas. However, it is possible to use it to keep larger groups of machines in sync by performing multiple pairwise synchronizations.

If you need to do this, the most reliable way to set things up is to organize the machines into a “star topology,” with one machine designated as the “hub” and the rest as “spokes,” and with each spoke machine synchronizing only with the hub. The big advantage of the star topology is that it eliminates the possibility of confusing “spurious conflicts” arising from the fact that a separate archive is maintained by Unison for every pair of hosts that it synchronizes.

Answer 3

How about to put your files in a version control system like SubVersion or git?

I don't version my entire home dir, but only a handful subdirs with some important stuff. But I am thinking about converting to git since it seem to be better for this type of job. (a quick search is going to send you in the right direction).

Good luck

---

Update: A nice side effect with git is that it is easy to have computer specific files, since you only need to have a branch for that computer (and in git you get that by default). So you don't need to have exactly the same files in all the computer, they only need to be similar, and have a common "main" branch with stuff that you want on all the different computers.

Answer 4

Your main problem when trying to do this is deciding how to merge changes, propagate deletes, and resolve conflicts. This is hard to do in a completely automated way, especially if you've got a 3 computer setup used by multiple people.

If you separate users things get much simpler. Because one user can't be in two places at once (and therefore generate conflicts) you could then setup an rsync job to run on login to "get changes" and logout to "push changes" ... to one of your computers which would be the master ... so at this level of granularity you'd be syncing /home/myuser each time rather than the entire /home/. An added refinement (in case of people not logging out) would be run the push script after a short period of inactivity.

There are lots of other potential solutions, but none that magically solve this problem as far as I'm aware. The first step is probably to think about how you use the machines and come up with a syncing policy to suit your user's behaviour.

Answer 5

I think you can achieve what you want better by NFS mounting a common home folder. Check out this article http://www.linuxjournal.com/article/4880

Answer 6

Check out lsyncd

Lsyncd watches a local directory trees event monitor interface (inotify or fsevents). It aggregates and combines events for a few seconds and then spawns one (or more) process(es) to synchronize the changes. By default this is rsync. Lsyncd is thus a light-weight live mirror solution that is comparatively easy to install not requiring new filesystems or blockdevices and does not hamper local filesystem performance.

Rsync+ssh is an advanced action configuration that uses a SSH to act file and directory moves directly on the target instead of retransmitting the move destination over the wire.

Fine-grained customizaton can be achieved through the config file. Custom action configs can even be written from scratch in cascading layers ranging from shell scripts to code written in the Lua language.layers ranging from shell scripts to code written in the Lua language.

Answer 7

This won't entirely provide a solution, but it will give you a start:

Set up a cron job every so often to rsync the files. I use a command like the following:

rsync -alhz --stats --progress --exclude-from '/etc/rsync_backup_excludes.conf' / -e ssh user@10.10.0.1:/path/to/home/

This will use rsync to only copy the needed changes, not re-copy everything every time the command is ran.

--stats and --progress are optional and should probably be excluded from your scripts

the contents of my rsync_backup_excludes.conf file are:

dev/*
proc/*
lost+found/*
mnt/*
sys/*

Answer 8

Take a look at dropbox http://www.getdropbox.com/ Cross platform Win / Linux / Mac

Answer 9

Mount /home from one computer over to the other two. Automount works pretty well for this.

Answer 10

Ubuntu One might be what you're looking for. Unfortunately it's still in beta phase, and I have no experience with it, so I'm not sure if it'd work for you.

Answer 11

If you also want to sync configs - tools mentioned above are totally helpless: configs are often changed, and many logfiles will make conflicts so they can't be merged.
My deceision is simple, and stupid :) I've got a Master workstation, and all files from "~" are just copied to the second "Slave" one. The moment i realize I need to modify something - i do it on Master, and Slave catches these changes on sync.

Also, some bash scripts should run different on these machines, so I edited my /etc/bash.bashrc:

export OOHOST=Master

Now, scripts know which host they're serving ;)

Answer 12

Try dvcs-autosync.

This is based on git, uses XMPP to notify conected clients of file changes and reacts on file change events via inode changes. So it always gets informed right after the file change, in case of conflicts it relies on the proven methods of git.

I've been using it for a while now as a Dropbox replacement via SSH public keys and it really does the job.

Answer 13

I use SyncTwoFolders for Mac and Syncback for PC - the first one for my three macs at home (inclusive laptop), and the second for my PC at work. Both program have various back-up and sync options making this a breeze once pre-sets have been set up. Easy-peasy! I'm sure there are other versions of similar software available for Linux and Ubuntu.

Answer 14

The workarounds I would consider:

• Dropbox.
  Main drawbacks: it's not free for any reasonable amount of data. Otherwise it works flawlessly IMHO

• A NAS server. Put your files on a NAS server, they became reasonably affordable lately (from 150 $ or so) and a good one doesn't consume much energy. No synching issues ever.
  Main drawback: slower than a local copy

Answer 15

PowerFolder - Sync Files, Sync Folders, Remote Storage, Backup and Private File Sharing. Sync home and office PC, share holiday pictures or work together on documents. PowerFolder's secure peer-to-peer technology works over the Internet or in LAN.

Answer 16

You could use a sync script like osync which can propagate deleted files and update only modified ones. http://www.netpower.fr/osync osync is rsync based but can handle a lot of sync scenarios between local folders or remote ones over ssh.