46

I have a directory of many files, something like 50,000 pdf's and other files on a server. I need to move specific ones to another directory. I can generate a list of the files that need to be moved either in csv or any other text format.

What I need to do is run a bash script and move or copy the files that are listed in the text file to another directory.

Is there an easy way of doing this? Any suggestions or resources would be greatly appreciated.

Jestep
  • 633
  • 1
  • 6
  • 11
  • Also, the files are named randomly when they are added to the directory. There is no naming convention or any logic that could be used to move specific files. – Jestep Dec 14 '10 at 14:09

10 Answers10

41

In order to avoid a useless use of cat (and if you don't use rsync):

xargs -a file_list.txt mv -t /path/to/dest

This will handle any valid filename, unless it contains a newline, if the files are listed one per line.

Dennis Williamson
  • 60,515
  • 14
  • 113
  • 148
  • 2
    One disadvantage of this approach is that if the files in the list contain path names (i.e. they are not all in the same directory), the use of `mv` will collapse all of them into the same destination directory. The `rsync` approach in the accepted answer doesn't suffer from this limitation. – Jason R May 04 '12 at 16:10
  • Also, I tried this command and xargs would break on any files that contained a single quote ("unmatched single quote") – James Beninger Aug 17 '12 at 01:30
  • This worked for me because I actually wanted the files from different source paths in the same target directory. – PseudoNoise Apr 29 '15 at 17:17
40

rsync has several options that can take a list of files to process(--files-from, --include-from, etc.).

For example, this will do the trick:

rsync -a /source/directory --files-from=/full/path/to/listfile /destination/directory
Vadim Kotov
  • 111
  • 4
Ignacio Vazquez-Abrams
  • 45,019
  • 5
  • 78
  • 84
  • 2
    example: ```rsync -a /source/directory --files-from=/full/path/to/listfile /destination/directory``` – anneb Oct 31 '18 at 14:04
  • 3
    If files list contains absolute paths, use `/` as source directory. – yahol Jun 26 '20 at 14:33
  • 2
    You also must use --no-relative if you don't want to keep the path, and have it be flat. – thouliha Jun 05 '21 at 16:02
  • 2
    -a normally implies -r, but *not* when using the "--files-from" option. For recursive behavior, you need to explicitly add the -r option also, as in "rsync -ar /source/director --files-from=/full/path/to/listfile /destination/directory" – pob Jun 08 '21 at 17:21
18
for file in `cat listoffiles`; do mv "$file" /path/of/destination ; done

assuming bash, plus listoffiles containing one file per line. My only objection to the rsync route is that the OP asks for a method of moving the files, not copying them. rsync has more flags than a royal wedding, so I'm sure it can be further modified to do that, but sometimes simpler is better!

MadHatter
  • 78,442
  • 20
  • 178
  • 229
  • 1
    This solutions strikes me as the most flexible. I ended up using it with rsync instead of cp or mv just because I needed the relative file paths option. – Spamwich Aug 06 '15 at 20:50
  • 5
    Nice, just I had problems with spaces, so I modified it to: `while read -r file; do mv "$file" /path/of/destination ; done < listoffiles` – user1182474 Feb 28 '16 at 14:19
  • 1
    @user1182474 Thanks, and an elegant use of input redirection. The traditional method of showing satisfaction with an answer is to upvote, by the way! – MadHatter Feb 28 '16 at 17:36
  • 1
    @user1182474 thank you for that! last step in many modifications had to somewhat blindly make to port my OSX backup utilities to my Android! – Charlie Gorichanaz Sep 15 '16 at 03:24
5
rsync --files-from=file_list.txt /path/to/source/ /path/to/dest/

Rsync has the added benefit over the cp or mv commands because it will automatically create folders if they do not exist.

Paul Wenzel
  • 151
  • 1
  • 3
4

This depends on the format of the text file you have. For example, if you have the list of files written such that Each file is located on a new line. You can use xargs like:

$ cat your_text_file | xargs cp -t /path/to/destination

Also, you can use find command with -exec option. to copy/move the files.

Khaled
  • 35,688
  • 8
  • 69
  • 98
2

If (and only if), you don't have any nasty characters in the filenames (spaces, newlines, which would confuse xargs on how to break things into individual arguments), and you generate a list of files separated by newlines (one file per line), you could do something like.

cat filenames.txt | xargs mv -t /path/to/move/files/to

(In general, see man xargs, it's awesome)

If your particular mv doesn't have the -t option, you could also do some trickery like

( cat filenames.txt; echo; echo /path/to/move/files/to ) | xargs mv

Note - neither of these will work as expected if there happens to be filenames with newlines in them.

Kjetil Joergensen
  • 5,854
  • 1
  • 26
  • 20
  • However; see Ignacio's answer, rsync is sort-of made for this stuff. – Kjetil Joergensen Dec 14 '10 at 14:21
  • `cp`, `mv` and the like do not have `-t` option on [tag:bsd]'s, and the second example with subshell did not work for me (`/path/to/move` is appended as the last filename) –  Jan 09 '19 at 17:04
  • 1
    @w17t - sounds like filenames.txt does not end with a newline, either add newline to the end of filenames.txt, or do something like `(cat filenames.txt; echo; echo /path/to/move/files/to) | xargs mv` – Kjetil Joergensen Jan 10 '19 at 23:38
2

I think the rsync answer is a better one, but just for another option:

tar -cf - -T FILE_OF_FILENAMES_TO_MOVE.txt |(cd /path/to/new/dir && tar -xvf -)
jj33
  • 11,038
  • 1
  • 36
  • 50
1

Try something like:

cat list.txt | while read line; do mv "$line" /images; done
Scott Pack
  • 14,717
  • 10
  • 51
  • 83
AliGibbs
  • 2,303
  • 20
  • 34
  • could have been `while read line; do mv $line /images; done < list.txt` as a single command – ignivs Sep 17 '15 at 19:12
1

I accidentally copied the full contents of a directory into a destination directory instead of moving the full directory. This resulted in a cluttered destination directory instead of the origin directory being added to the directory.

To fix this, I did the following:

ls -rt /path/to/cluttered/destination/directory/ > /opt/dircheck/filestomove The above command creates the filestomove file that will be a list of all contents of the destination directory, reverse sorted by time, meaning oldest to newest.

Then I created a sub-directory of the now cluttered destination directory to move the stuff into.

mkdir /path/to/cluttered/destination/directory/newsubdirectory

Then I repeated the directory listing, except listing to screen and showing more details.

ls -lrht /path/to/cluttered/destination/directory/ This line lists the directory, sorted by date ascending (reverse sort by time) and shows more information, including the date/time stamp of each file in the now cluttered destination directory. I refer to this, starting at the top to show what directories and files I want to keep where they were. There will be a gap in the date/time stamp of the files where all the new files start that shouldn't be there.

Then I edited the filestomove file created in the first step above (that is sorted by date) and deleted the few from the list that were there previously that I want to stay from the original directory.

vim /opt/dircheck/filestomove Delete from the top, all files you don't want to move.

Then I used the command listed previously in this post to move the files in my list to the new directory I made.

sudo xargs -a /opt/dircheck/filestomove mv -t /path/to/cluttered/destination/directory/newsubdirectory

This moved all the files in a split second. (Note: you may not need the sudo at the beginning, this is a matter of file permissions).

Now my original destination directory is nice and clean and contains a new directory with all the files and directories that were cluttering it.

John
  • 11
  • 1
0

The following worked for me where I needed to copy all the PNG files from a specific path (and all subdirs) to a new location while preserving the directory structure:

rsync -av --prune-empty-dirs --include='*/' --include='*.png' --exclude='*' source/ destination/

Because RSYNC creates the mirror of the directory structure first, then syncs the files, you can end up with excess folders that are empty. I used the --prune-empty-dirs flag to remove these empty directories.

I have no affiliation, but thought it right to give credit to the source that inspired this solution: http://techblog.zabuchy.net/2011/transfer-only-selected-file-types-with-rsync/

slm
  • 7,355
  • 16
  • 54
  • 72
John Mark Mitchell
  • 196
  • 4