1

I am using a local website which downloads files to my server. Using an Inotify daemon I'm listening for file changes in a specific folder (the one I am downloading to) and executing a script which moves those files into another folder using rsync. However my problem here is that rsync moves the files before they are completely downloaded so they are at 0KB. I used mv before, and mv somehow managed to wait until the download is completed before moving, but I needed to use rsync because mv won't write to existing, non-empty folders.

Any ideas?

Here is my script which executes on inotify create:

#!/bin/sh

cd the_folder_which_is_watched_by_inotify

rsync -vh * /where_the_files_are_moved_to

echo "Files have been moved"
PoTTii
  • 21
  • 1
  • 4
  • Not sure what you mean by this `mv won't write to existing, non-empty folders`. – Zoredache May 25 '16 at 17:15
  • @Zoredache MV returns `mv: cannot move './dir' to './dir2': Directory not empty` when writing to folders containing files – PoTTii May 25 '16 at 18:14
  • Why are you trying to move the folder, instead of moving the file? – Zoredache May 25 '16 at 18:17
  • Because my download tool bulk downloads files and creates multiples folders with those files in it. – PoTTii May 25 '16 at 18:20
  • Ok, but why would your inotify command try to move the folders instead of the individual files? Can you setup inotify to ignore folders and only work with the individual files? – Zoredache May 25 '16 at 19:24

2 Answers2

1

The reason mv would probably work, is because the file would actually be the same file. So the upload process would be writing to a given inode, and the mv command simply updates the source and destinations so that the inode is in a different location. But any programs with an open handle to the file would be able to continue working with that inode, so you can move the link around to wherever you want.

What you might be able to do is employ the rsync --link-dest option. Getting the paths right is tricky since --link-dest needs the relative path to the source from the perspective of the destination. But it should result in rsync making hardlinks to the file. That is both the source and destination should link to the exact same inode.

Community
  • 1
Zoredache
  • 128,755
  • 40
  • 271
  • 413
1

You should setup your inotify daemon to launch rsync only on CLOSE_WRITE and MOVE_FROM/MOVE_TO combo events.

If you are listening for CREATE and/or MODIFY events, you will end with rsync called when the file is being written to, which will led to a corrupt/empty copy.

shodanshok
  • 44,038
  • 6
  • 98
  • 162
  • Great, I am going to try that out later, thank you. What do you mean by `MOVE_FROM/MOVE_TO` combo events? – PoTTii May 26 '16 at 07:59
  • When moving/renaming a file, the inotify subsystem generate two events: a MOVE_FROM (with the original file name/path) and a MOVE_TO (with the new name/path). These two events are emitted as two separate events, but they really form a single, combined event that has to be reconstructed using the `coockie` value embedded in these events. For more information, `man inotify` is your friend ;) – shodanshok May 26 '16 at 09:15