rsync - does it create a temp file during transfer?

39

5

As far as I can see rsync doesn't create the file in the target directory until it is complete.

This must mean that it creates the file in a temp directory somewhere and copies the file into the target directory when it is complete.

First off, is this correct?

If true, is it possible for rsync to set not use a temp directory and instead create the file in the target directory and just keep writing to it until it's complete?

Cheetah

Posted 2013-04-06T18:17:48.857

Reputation: 891

Answers

48

Yes, it does create a temp file.

According to the man page you can specify the directory in which the temp file is stored with the parameter -T as in

-T, --temp-dir=DIR create temporary files in directory DIR

The answer to the second part of your question can also be found there:

--inplace

This option changes how rsync transfers a file when the file's data needs to be updated: instead of the default method of creating a new copy of the file and moving it into place when it is complete, rsync instead writes the updated data directly to the destination file.

superuser0

Posted 2013-04-06T18:17:48.857

Reputation: 889

4Important note! The question assumes rsync "copies the file into place", whereas the rsync documentation specifies that rsync is actually "moving it into place", which is an IMORTANT DISTINCTION. This ensures that the file appears in the destination location in an atomic fashion. This is useful for preventing race conditions where another application is waiting for the destination file to appear and immediately opens it for reading, but the file is not done being copied. – Ogre Psalm33 – 2017-01-12T16:08:04.223

15

It does create a temp file, by default in the target directory and named .<FILE_NAME>.<RANDOM_STRING>. So, if you are copying foo.txt, it will create a tmp file called .foo.txt.GV4H3 (GV4H3 is the random string that will be different every time you run it). You can control this behavior using these rsync options:

   --partial
          By default, rsync will delete any partially transferred
          file if the transfer is interrupted.  In  some  circum‐
          stances  it  is more desirable to keep partially trans‐
          ferred files. Using the --partial option tells rsync to
          keep  the  partial  file which should make a subsequent
          transfer of the rest of the file much faster.

   --partial-dir=DIR
          A better way to keep partial files than  the  --partial
          option  is  to  specify a DIR that will be used to hold
          the partial data (instead of writing it out to the des‐
          tination file).  On the next transfer, rsync will use a
          file found in this dir as data to speed up the  resump‐
          tion  of  the  transfer and then delete it after it has
          served its purpose.

Please read the relevant parts of the rsync man page (the following is only a small extract of the large section on how to use --partial-dir).

terdon

Posted 2013-04-06T18:17:48.857

Reputation: 45 216

5

rsync creates temporary files in the target directory. That file is named like the source but with an extension. It could be like that:

source: foo.bar 
target temp: foo.bar.hzmkjt7 (hzmkjt7 is just an example) 

The extension is removed after the file has been verified to be an exact copy of the source. During the renaming process ownership, permissions and modification time are set. So it is possible that you don't see the file because of the permissions. Make sure that you have enough permission to see all the files as well those not owned by you in order to see the temp files.

Simon

Posted 2013-04-06T18:17:48.857

Reputation: 3 831

A process creates files with the permissions of the user who runs it. It is not possible to not see the files because of permissions in this case. – Ярослав Рахматуллин – 2013-04-06T22:00:50.860

@ЯрославРахматуллин rsync can as well be run in daemon mode. Or it can be scheduled as a cronjob. Either way it is possible that the user under whom rsync is running is not the user who wants to see the temp file. In that case it is possible that it is a permission issue. – Simon – 2013-04-07T12:51:20.927

1The temp file might start with ".". In this case, it is .foo.bar.hzmkjt7. – interskh – 2013-06-28T20:35:29.987