29

From time to time, I have to perform several large migration changes on data files on my server, and I'm looking for a good way to do this. I was thinking about using rsync to duplicate my directory structure starting at the root data folder, creating hard links to all original the files (some of them are rather big), and I can overwrite in the destination tree only the files that need migrating. In the end, I can safely switch from the old files to the new files with two mv operations.

However, I can't seem to get rsync to do this. I tried

rsync -a --link-dest=$DATA $DATA $DATA/../upgrade_tmp

but instead of creating hard links to files, rsync copies them entirely. Is there a problem using the same source and link-dest directory?

Sean Reifschneider
  • 10,370
  • 3
  • 24
  • 28
Jean-Philippe Pellet
  • 482
  • 1
  • 4
  • 11

8 Answers8

27

rsync is a powerful tool, but it is, unfortunately, strangely picky about some of its pathnames.

If $DATA is an absolute path (i.e. it begins with a /), then the correct command-line to use is:

rsync -a --link-dest=$DATA $DATA/ $DATA/../upgrade_tmp

[Now, just a brief aside about rsync's strangeness. Note the trailing / added to the source argument. This tells rsync to work with the contents of the source directory, rather than with the source directory itself. (I'm assuming that $DATA doesn't already contain a trailing /.) In this case, we want to work with the contents, so we add the trailing /.]

If, on the other hand, $DATA is a relative path (i.e. it does not begin with a /), then Sean R's comment about --link-dest is bang on: The link-dest path is interpreted relative to the destination path, so you would use the following:

rsync -a --link-dest=../`basename $DATA` $DATA/ $DATA/../upgrade_tmp

EDIT

One final note, it turns out that the second rsync command-line I gave should work regardless of whether $DATA is an absolute path, since basename doesn't care whether a path is absolute or relative.

Steven Monday
  • 13,019
  • 4
  • 35
  • 45
  • 1
    Just a missing slash, who would have thought of that… Thanks for the nice explanation! – Jean-Philippe Pellet Dec 08 '10 at 09:00
  • Thank you for this, I tried following several incremental backup instructions, like [this one](http://www.interlinked.org/tutorials/rsync_time_machine.html) and did not find any mention of this quirk. This was the only thing that actually ensured files were being hardlinked. Checking that ref count was > 1 and inode numbers were the same with `ls -ilah` – Walf Nov 12 '13 at 05:11
  • I used this in conjunction with the relpath() function described at http://unix.stackexchange.com/a/85068/57414 to backup a `$SOURCE` directory into a `$TARGET` dir like this: `SOURCE='abs_path_to_backup'; TARGET='.'; rsync -a --link-dest=$(relpath $TARGET $SOURCE) $SOURCE/ $TARGET/` – Nathan S. Watson-Haigh Jul 02 '14 at 03:30
18

What you want is "cp -al":

cp -al $DATA/ $DATA/../upgrade_tmp/
  • -a recurses like rsync -a
  • -l will hard link files instead of copying them.
Sean Reifschneider
  • 10,370
  • 3
  • 24
  • 28
9

Turns out it is more difficult to do this with rsync than with other tools. The correct answer for rsync is Steven Monai's, but the easiest way to do this is to use either cp -al or pax -rwl on systems where -l is not a valid option for cp:

pax -rwl $DATA $DATA/../upgrade_tmp

or

cp -al $DATA/ $DATA/../upgrade_tmp/
Jean-Philippe Pellet
  • 482
  • 1
  • 4
  • 11
7

The --link-dest option in rsync is relative to the destination directory, not the current directory. So what you want is:

rsync -a --link-dest=../`basename $DATA` $DATA $DATA/../upgrade_tmp
Sean Reifschneider
  • 10,370
  • 3
  • 24
  • 28
4

It works for me:

$ rsync --hard-links --recursive --link-dest=/local user@host:/remote/ /local

I use rsync version 3.1.0.

From man:

--hard-links

Tells rsync to look for hard-linked files in the transfer, without this option, hard-linked files in the transfer are treated as though they were separate files.

--link-dest=DIR

Unchanged files are hard linked from DIR to the destination directory. The files must be identical in all preserved attributes (e.g. permissions, possibly ownership) in order for the files to be linked together

peterh
  • 4,914
  • 13
  • 29
  • 44
Alexander Fedorov
  • 141
  • 4
  • 2
    Only code snippet is not enough, explain what it does and why. – peterh Apr 24 '15 at 17:39
  • **--hard-links** Tells rsync to look for hard-linked files in the transfer, without this option, hard-linked files in the transfer are treated as though they were separate files. **--link-dest=DIR** Unchanged files are hard linked from DIR to the destination directory. The files must be identical in all preserved attributes (e.g. permissions, possibly ownership) in order for the files to be linked together. – Alexander Fedorov Apr 30 '15 at 14:12
  • 1
    Wonderful. Thank you. Actually, I found your answer in the "low quality" queue. It means, there was a vote if your answer should be deleted or not. But not only the danger of the deletions is a reason to try to give a well-formatted, "human" answer, but it also helps a lot if you want to collect a upvotes. – peterh May 01 '15 at 02:00
2

Can try following link http://www.lessfs.com/wordpress/ it is work on COW (copy on write) that will save time and space

Community
  • 1
Rajat
  • 3,329
  • 21
  • 29
  • lessfs is very interesting, but is also very experimental. Not recommended for production use just yet. – mattdm Dec 07 '10 at 17:13
2

First create the directories only on the destination:

rsync -av --include '*/' --exclude '*' /source/ /destination/

Then hard link the files only:

cd /source
find . -type f -exec ln -v {} /destination/{} \;
Cakemox
  • 24,141
  • 6
  • 41
  • 67
1

Use the Option -H to preserve Hardlinks and read the manpage.

tex
  • 889
  • 1
  • 9
  • 19
  • 1
    -H doesn't work. I have no hardlinks to preserve in my source tree, I just want a simple copy of my source tree where in the copy, all files are hardlinked to the original files. Sorry in my original question was unclear… – Jean-Philippe Pellet Dec 07 '10 at 10:36
  • is "read the manpage" an answer? :-) – meduz Nov 01 '15 at 11:17