Get list of transferred files from rsync?

Question

I'm currently using rsync on a script that deploys a PHP application from a staging to a production server. Here is how:

rsync -rzai --progress --stats --ignore-times --checksum /tmp/app_export/ root@app.com:/var/www/html/app/

This is currently outputting a list of every file that's being compared (every file in the project), but I'd like it to output only the modified ones, so I can run it with a --dry-run option to check that every deploy is updating only the desired files.

NOTE: The best I could do so far is grep fcst the results, but I'm looking for an rsync option that I'm sure it's there but I can't find it in the man pages.

Thanks in advance!

score 10 · Answer 1 · answered Jul 20 '17 at 12:32

Use the `--out-format` option

According to the man page:

Specifying the --out-format option will mention each file, dir, etc. that gets updated in a significant way (a transferred file, a recreated symlink/device, or a directory).

If you only need the actual filenames (--out-format="%n") your dry run command might look like:

rsync -rzan --out-format="%n" --ignore-times --checksum /tmp/app_export/ root@app.com:/var/www/html/app/

When rsync is called with -v, it internally uses this option with a default format of "%n%L", which tells you just the name of the file and, if item the is a link, where it points.

But this also includes a short summary at the beginning and the end of the sync process.

To get rid of that summary use the --out-format option directly.

Btw. -i also internally uses --out-format, but with a format of "%i %n%L".

score 8 · Answer 2 · answered May 17 '15 at 10:22

Beginning with rsync v3.1.0, released in 2013, there is the --info flag which allows fine-grained control over the output.

 --info=FLAGS
          This option lets you have fine-grained control over the information output you want to see.  An individual flag name may be followed
          by a level number, with 0 meaning to silence that output, 1 being the default output level, and higher numbers increasing the output
          of that flag (for those that support higher levels).  Use --info=help to see all the available flag names,  what  they  output,  and
          what flag names are added for each increase in the verbose level.  Some examples:

              rsync -a --info=progress2 src/ dest/
              rsync -avv --info=stats2,misc1,flist0 src/ dest/

          Note  that  --info=name’s  output  is  affected  by the --out-format and --itemize-changes (-i) options.  See those options for more
          information on what is output and when.

          This option was added to 3.1.0, so an older rsync on the server side might reject your attempts at fine-grained control (if  one  or
          more  flags  needed  to  be  send to the server and the server was too old to understand them).  See also the "max verbosity" caveat
          above when dealing with a daemon.

The available --info flags are:

Use OPT or OPT1 for level 1 output, OPT2 for level 2, etc.; OPT0 silences.

BACKUP     Mention files backed up
COPY       Mention files copied locally on the receiving side
DEL        Mention deletions on the receiving side
FLIST      Mention file-list receiving/sending (levels 1-2)
MISC       Mention miscellaneous information (levels 1-2)
MOUNT      Mention mounts that were found or skipped
NAME       Mention 1) updated file/dir names, 2) unchanged names
PROGRESS   Mention 1) per-file progress or 2) total transfer progress
REMOVE     Mention files removed on the sending side
SKIP       Mention files that are skipped due to options used
STATS      Mention statistics at end of run (levels 1-3)
SYMSAFE    Mention symlinks that are unsafe

ALL        Set all --info options (e.g. all4)
NONE       Silence all --info options (same as all0)
HELP       Output this help message

Options added for each increase in verbose level:
1) COPY,DEL,FLIST,MISC,NAME,STATS,SYMSAFE
2) BACKUP,MISC2,MOUNT,NAME2,REMOVE,SKIP

This is awesome, thanks. I had no idea this flag existed because I use a Mac - even the latest MacOS installs an rsync version from 2004 lol. Cheers Apple... — twistedpixel, Feb 05 '18 at 15:00
Thank you. To get the actual list of files, I used --info=NAME. — Yaroslav Nikitenko, Feb 07 '21 at 16:47

score 7 · Accepted Answer · answered Sep 11 '12 at 22:11

7

If there's an rsync option to do exactly what you're asking, I didn't find it in the manpage either. :-)

That said, I don't see the problem with grepping the output of rsync -i to parse out exactly what you need. That feels nice and Unixy to me.

One nit-picky quibble with your rsync command: the -r is redundant, as it is implied by -a.

answered Sep 11 '12 at 22:11

Kevin DeGraaf

456
3
4

2

Thanks man. I wonder why (with the variety of options this program has) there isn't an option for that. Seems pretty basic for me. – Mauro Sep 11 '12 at 22:15
Hi Kevin, actually there is an option: `-P` for progress does exactly that in the `-n` dryrun setting, although `-i` isn't wrong either. See my answer below. – Robert Riedl Jan 31 '19 at 08:20

score 1 · Answer 4 · answered Jan 31 '19 at 08:18

Actually, you kind of answered your own question, since your original command already has it: --progress

This is the correct option, although the man page is a bit cryptic about it:

     --progress              show progress during transfer
 -P                          same as --partial --progress

It kinda makes sense, since you invoke your rsync string with dryrun mode, no transer happens, but you'd still have progress: namely the files that have changed and would be transferred.

This way you get a neat little list of all the files, for example:

The destination already has a copy of changedfile which was updated in the source and oldfile, which remains unchanged. The source has also an additonal file: newfile.

#~$ ls -lhan /tmp/destination/
total 20K
drwxrwxr-x  2 1000 1000 4,0K Jän 31 09:07 .
drwxrwxrwt 18    0    0  12K Jän 31 09:15 ..
-rw-rw-r--  1 1000 1000    2 Jän 31 09:08 changedfile
-rw-rw-r--  1 1000 1000    0 Jän 31 09:07 oldfile



#~$ ls -lhan /tmp/source/
total 20K
drwxrwxr-x  2 1000 1000 4,0K Jän 31 09:07 .
drwxrwxrwt 18    0    0  12K Jän 31 09:15 ..
-rw-rw-r--  1 1000 1000    2 Jän 31 09:15 changedfile
-rw-rw-r--  1 1000 1000    0 Jän 31 09:07 newfile
-rw-rw-r--  1 1000 1000    0 Jän 31 09:07 oldfile

If we then invoke your rsync command, but remove itemization -i and just add dryrun -n

#~$ ~$ rsync -n -rza --progress --stats --ignore-times --checksum /tmp/source/ /tmp/destination/
sending incremental file list
changedfile
newfile

Number of files: 4 (reg: 3, dir: 1)
Number of created files: 1 (reg: 1)
Number of deleted files: 0
Number of regular files transferred: 2
Total file size: 2 bytes
Total transferred file size: 2 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 187
Total bytes received: 22

sent 187 bytes  received 22 bytes  418.00 bytes/sec
total size is 2  speedup is 0.01 (DRY RUN)

You get a list of only the files rsync would tranfser: changedfile and newfile.

score 0 · Answer 5 · answered Nov 28 '20 at 17:28

0

One note that applies globally so putting it as an answer -- make sure you are running at least rsync 3.0, or you'll be 15+ years out of date (default on Mac's).

brew update rsync

answered Nov 28 '20 at 17:28

pfrank

101
1

Willis · Answer 6 · 2022-02-20T10:57:11.223

Method 1

You can use --info=NAME option(only versions later than rsync 3.1.0 are supported)

rsync -azPrui --info=NAME /path/from/source/ /path/to/destination/

use rsync --info=help to get the value list of --info

Use OPT or OPT1 for level 1 output, OPT2 for level 2, etc.; OPT0 silences.

BACKUP     Mention files backed up
COPY       Mention files copied locally on the receiving side
DEL        Mention deletions on the receiving side
FLIST      Mention file-list receiving/sending (levels 1-2)
MISC       Mention miscellaneous information (levels 1-2)
MOUNT      Mention mounts that were found or skipped
NAME       Mention 1) updated file/dir names, 2) unchanged names
PROGRESS   Mention 1) per-file progress or 2) total transfer progress
REMOVE     Mention files removed on the sending side
SKIP       Mention files that are skipped due to options used
STATS      Mention statistics at end of run (levels 1-3)
SYMSAFE    Mention symlinks that are unsafe

ALL        Set all --info options (e.g. all4)
NONE       Silence all --info options (same as all0)
HELP       Output this help message

Options added for each increase in verbose level:
1) COPY,DEL,FLIST,MISC,NAME,STATS,SYMSAFE
2) BACKUP,MISC2,MOUNT,NAME2,REMOVE,SKIP

Method 2

You can use -i option to get the new added or updated files to destination folder(all versions are supported)

rsync -azPrui /Users/bruce/Code/test1/ /Users/bruce/Code/test2/ | awk '/>f+++++++|>f.st......./{print $2}'

Method 3

Use --out-format="%n" option(all versions are supported)

rsync -azru --out-format="%n"  /Users/bruce/Code/test1/ /Users/bruce/Code/test2/

To see all the format, you can use man rsyncd.conf command and search log format in it.

Nux · Answer 7 · 2020-10-09T13:40:33.223

0

Not sure if this differs between versions/options used but in my version when I use -i option I get a list like:

>f..T...... existing-file.png
>f+++++++++ new-file.png
cd+++++++++ new-dir/
>f+++++++++ new-dir/new-file.png

So a simple solution to get only a list of files actually transfered just run:

rsync [your options here] | grep -v "f..T......"

This will simply hide all lines containing f..T....... So effectively this will hide identical files.

edited Oct 09 '20 at 13:40

answered May 27 '16 at 16:18

Nux

541
3
12
21

in order not to show the changed files that have changed timestamp omit timestamp with -I so -i -I – aimiliano Oct 09 '20 at 11:59

Get list of transferred files from rsync?

7 Answers7

Use the --out-format option

Method 1

Method 2

Method 3

Use the `--out-format` option