6

Sometimes I have two trees that used to have the same content, but have grown out of sync (because I moved disks around or whatever). A good example is a tree where I mirror upstream packages from Fedora.

I want to merge those two trees again by moving all of the files from tree1 into tree2.

Usually I do this with:

rsync -arv tree1/* tree2

Then delete tree1.

However, this takes an awful lot of time and disk space, and it would be much easier to be able to do:

mv -r tree1/* tree2

In other words, a recursive move. It would be faster because first of all it would not even copy, just move the inodes, and second I wouldn't need a delete at the end.

Does this exist ?

As a test case, consider the following sequence of commands:

$ mkdir -p a/b
$ touch a/b/c1
$ rsync -arv a/ a2
sending incremental file list
created directory
./
b/
b/c1
b/c2

sent 173 bytes  received 57 bytes  460.00 bytes/sec
total size is 0  speedup is 0.00
$ touch a/b/c2

What command would now have the effect of moving a/b/c2 to a2/b/c2 and then deleting the a subtree (since everything in it is already in the destination tree) ?

Thomas Vander Stichele
  • 1,055
  • 4
  • 14
  • 16
  • this is a very good question. where i work ppl have solved the problem using some complex bash and awk scripts – biosFF Jun 11 '09 at 20:45
  • My preferred inefficient approach is `cp -R tree1/* tree2` followed by deletion of tree1. – RomanSt Aug 08 '11 at 13:02

11 Answers11

2

Per the mv(1) manpage from gnu's mv:

-u, --update move only when the SOURCE file is newer than the destination file or when the destination file is missing

Bill B
  • 399
  • 5
  • 12
  • 2
    Are you sure this works ? For example; [thomas@ana ~]$ mkdir -p a/b [thomas@ana ~]$ touch a/b/c1 [thomas@ana ~]$ rsync -arv a a2 sending incremental file list created directory a2 a/ a/b/ a/b/c1 sent 125 bytes received 39 bytes 328.00 bytes/sec total size is 0 speedup is 0.00 [thomas@ana ~]$ touch a/b/c2 [thomas@ana ~]$ mv -u a/* a2 mv: cannot move `a/b' to `a2/b': Directory not empty [thomas@ana ~]$ mv -u -T a a2 mv: cannot move `a' to `a2': Directory not empty – Thomas Vander Stichele Jun 12 '09 at 09:42
2

the proposed mv -uf dir1/* dir2/ move the (sub)directories, not each file. you might try to use find

cd dir1
find . -type d -exec mkdir -p dir2/"{}" \;
find . -type f -exec mv -uf "{}" dir2/"{}" \;

or something similar

Javier
  • 9,078
  • 2
  • 23
  • 24
1

Does not

mv -uf tree1/* tree2/

work?

David Spillett
  • 22,534
  • 42
  • 66
1

Midnight Commander (mc) is also nice for this kind of stuff. Tag files with CTRL-t, press F6, and when it asks to overwrite destination files, choose Update if you want to overwrite older files.

sunny256
  • 861
  • 7
  • 8
1

You could use "cp -l & rm" for in-device moving:

cp -alv --backup=numbered tree1/* tree2 &&
rm -rf tree1/
  • -l of cp to use hard links instead of copying, (this also prevents cross-device operations)
  • --backup=numbered of cp for backing up the existing files in target directory

And be careful about this two issues:

  • use && to prevent your uncopied data to be removed, if you accidentally run it on cross-device targets. (in corss-device case cp exits with a status of "1", at least for GNU coreutils)
  • files starting with "." in tree1, you will lost them if there is any.
Bekir Dogan
  • 111
  • 2
  • cp is terrible at this sort of thing because it typically destroys hard links / soft links and replaces them with individual files. Similarly it can't be trusted to keep ACLs straight and sometimes gets confused about sparse files. – chris Jun 28 '13 at 17:57
1

Javier's answer with find works well, except that it doesn't remove the original directories. Add at the end:

rmdir $(find . -type d  |grep -v ^\.$)
Howie Goodell
  • 11
  • 1
  • 1
    Howie, Welcome to ServerFault! Since your answer adds on to someone else's it might be best left as a comment to their answer. Or, you can edit your answer to fulfill the question in it's entirety. (If his question were deleted, yours should stand on it's own.) – Aaron Copley Jun 28 '13 at 17:44
  • This fails if directory names contain whitespace. Better: `find . -type d -not -name '.' -delete` (or `!` instead of `-not`) – Dennis Williamson Jul 08 '13 at 21:54
0

None of the answers in this thread fit my usecase, so I came up with one on my own as a shell script.

At the heart of it is this function:

recurse() {
    if [ ! -d "$1" ] || [ ! -e "$2" ]; then
        mv -u "$1" "$2" || exit
        return
    fi
    for entry in "$1/"* "$1/."[!.]* "$1/.."?*; do
        if [ -e "$entry" ]; then
            recurse "$entry" "$2/${entry##"$1/"}"
        fi
    done
}

Which you could invoke like so (for the OP's usecase):

recurse a/b/c2 a2/b/c2

If source is a file or a directory for which the destination does not exist yet, it simply moves it with mv -u. If source is a directory for which the destination already exists, it iterates through its contents and recursively performs the same check followed by move|recurse with each of its members.

I used -u because I only needed old files freshened and newer files untouched, but you could replace it with -f for an unconditional move or -i for an interactive one. Or not change anything and simply rm -rf your source after the script is done moving things.

antichris
  • 101
  • 2
0

err

mv dir1/* dir2/

or simply

rsync -arv --remove-source-files  tree1/* tree2

should be enough, you will probably run into trouble at some point when too many entries are in dir1.

find sourcedir -maxdepth 1 -exec echo mv {} targetdir/ \;

should be a nice options

find sourcedir -maxdepth 1 -print0 |xargs -0 -I _ echo mv _ targetdir
find sourcedir -maxdepth1 -exec mv {} targetdir/ +

are both not really necessary because mv just takes 2 options (source target), so you will have to live with the multitude of processes in that case.

Martin M.
  • 6,428
  • 2
  • 24
  • 42
  • This is not what I'm asking for. Your solution goes only one level deep and doesn't recurse into subdirectories. – Thomas Vander Stichele Jun 12 '09 at 09:48
  • It doesn't recurse because with the subsequent mv command it's not necessary to recurse any further than one directory. However since you started with rsync I recommend my the --remove-source-files switch which will keep you with the tool you know plus it won't use as much space as the rsync/rm combination. – Martin M. Jun 12 '09 at 10:02
  • On another note: since I just read your post again regarding the merging you might want to play with -n or -i options from mv – Martin M. Jun 12 '09 at 10:03
  • I'm pretty sure you're not considering the fact that most of the tree already exists in the destination, so your first mv command will not work. Not sure what -n or -i will help here. Please try the example. – Thomas Vander Stichele Jun 13 '09 at 07:41
0
cd /tree1
mv * /tree2

This won't move hidden files or folders, but your original example wouldn't either.

Dave Cheney
  • 18,307
  • 7
  • 48
  • 56
0

I think mv doesn't do what you think it does.

A unix filesystem has 3 components:

  • directory entries
  • inodes
  • blocks

A directory entry points to an inode.

The inode has the metatadata about the file (is it a file, a directory, a named pipe? Who owns it? What are the permissions? What blocks does that inode use?

Blocks are the things that actually contain the contents of the file.

So -- when you "mv" a file, all you're really doing is unlinking the first directory entry and relinking it somewhere else.

snoopy -> inode 333 
woodstock -> inode 333

No data ever gets duplicated / copied. You create the link snoopy, then you create the link woodstock, and then you delete the link snoopy. (things are a bit different with directories because typically you can't make hardlinked directories, but even so the "link" name just changes).

What if you're moving from one filesystem to another? In the old days, mv would just thrown an error and make it explicit that you can't move a file from one filesystem to another. These days, it seems like mv silently copies the data then deletes the original.

In the old days, because you couldn't move data from one file system to another, you got in the habit of using idioms like

tar -cf - . | (cd /new/location && tar -xf -)

then you delete the old data. Part of the reason for using tar was that in the old days, cp would destroy metadata like "this is a symbolic link" and "that's a hard link" and instead you'd just get new copies of that file as regular files. Even still, you need to give "cp" flags to tell it to preserve that sort of structure.

There is no way to avoid "moving" lots of data if it is from one filesystem to another. It doesn't matter if you're using fancy new move or rsync or tar or cpio.

But, if you keep all the data in the same filesystem, this:

mv /filesystem-1/big/directory* /filesystem-1/big2/

that will be extremely fast because it is just changing the directory entries and not actually moving any real data.

There are other issues at play, such as what should you do if there is already a file / directory in the new location as well as the source location?

chris
  • 11,784
  • 6
  • 41
  • 51
-1

switch to directory you want to move and do

tar cf - * | ( cd /target; tar xfp -)

Quicker than mv...

bsdjunkie
  • 121
  • 5