Is there a way to tell if a file is done copying?

Question

The scenario is this: Machine A has files I want to copy to Machine C. Machine A can't access C directly, but can access Machine B that can access Machine C. I am using scp to copy from Machine A to B, and then from B to C.

Machine B has limited storage space, so as files come in, I need to copy them to C and delete them from B. The second copy is much faster, so this is no problem with bandwidth.

I could do this by hand, but I am lazy. What I would like is to run a script on B or C that will copy each file to C as each one finishes. The scp job is running from A.

So what I need is a way to ask (preferably from a bash script) if file X.avi is "done" copying. Each of these files is a different size, and I can't really predict size or time of completion.

Edit: by the way, the file transfer times are something about 1 hour from A to B and about 10 minutes from B to C, if time scale matters at all.

Answer 1

A common way of doing this is to first copy it to a temporary file name, preferably a hidden file. When the copy finishes, the script that is doing the copy then renames it to the non-hidden filename.

The script on machine B could then watch for non-hidden files.

The script on machine A would look something like this:

for file in `ls *` ; do
    scp $file user@host:~/.${file}.tmp
    ssh user@host "mv ~/.${file}.tmp $file"
done

Although this does not satisfy OP's desire to use the one-line

scp * user@host:~/

it accomplishes the same thing and also allows machine B to transfer each file as it finishes without waiting for the next file.

Answer 2

Does lsof on machine B show that scp has the file open? if so, you could watch lsof and see when scp closes the file. If not, you could watch the size of the file and after it hasn't changed for a given period of time (5 minutes, for example), copy it from B to C.

A third option would be to copy the files from A to to an "in_progress" directory on C. After the copy finishes on A, execute a mv command to move out of the "in_progress" directory.

Answer 3

I just thought of another, completely unrelated option. Doesn't use scp at all. Please let me know if this would work:

1. on B, create a fifo pipe somewhere: mkfifo /tmp/xfer

2. on A, don't use scp, instead, tar -cz files | ssh B 'cat > /tmp/xfer

3. on C, run ssh B 'cat /tmp/xfer' | tar -xz

This way, data isn't stored on B, it just passes through the pipe. The downside to this is, you can only have one copy going at a time...

You'll need to make sure the process on C respawns each time it finishes.

Answer 4

After thinking about the answers posted (in particular @Josh's idea of watching modified times) I was trying to run manipulate B's files on C. See, B is anaemic as far as available tools, so nothing that seemed to be able to do the job was there. I came upon this solution. This idea is not mine, I found it in google searches before this question. I discarded it earlier, since machine B did not have the find utility.

First, mount the appropriate directory on B onto C, so it appears as a local file system. I used sshfs for this (awesome tool, by the way). This will let me use C's utilities instead of B's.

Secondly, the command find /the/folder/* -mmin +5 will match all files modified over 5 minutes ago. So the command find /the/folder/* -mmin +5 -exec {} /the/other/folder \; will move all files that have been modified over 5 minutes ago to the other folder (which is actually on C, instead of sshfs mounted from B.

Finally, I set up a cron script to run the above script every 10 minutes today and tomorrow. The line in my crontab looks like this.

*/5 * 22,23 9 * find /the/folder/* -mmin +5 -exec mv {} /the/other/folder \;

Hopefully this will work. The next file has yet to complete, so I can't comment on if it really works when combined with the cron script, but I made some files by hand and seeded them and they moved fine. cross my fingers

Edit: This is working, though how it was originally had some errors, those are corrected now.

Answer 5

No need for mkfifo. On machine B, run this:

ssh A 'tar -cz files' | ssh C 'tar -xz'

You might find tar's -C option useful.

If you need to initiate the copying on machine A, just do:

tar -cz files' | ssh B "ssh C 'tar -xz'"

Watch out for proper quoting, though.

Answer 6

The copy will either run as a another process, or you could force it to, using a subshell. Then, you could use ps to "watch" the process and see when it disappears.

Also, I believe that in *nix, you can delete the file while it's being copied. The system won't delete it until the copy program closes it. Of course, if the copy doesn't succeed, you lose the file, so, not the best idea.