Is mv with wildcard still atomic

Question

My understanding is that mv dir1/file1 dir2/ is atomic,

Is mv dir1/* dir2/ also atomic?

As an example, assume there are 10 files in dir1 that are 10GB each.

As long as it is a move within one file system, the size of the files to move should be irrelevant (only the directory files concerned are changed). — Marc van Leeuwen, Mar 18 '15 at 08:32

roaima · Accepted Answer · 2020-01-24T11:06:53.170

22

Let's start with the statement that mv is not always atomic.

Let's also identify that atomicity refers to file contents, not to the file name.

For any individual file, the move or rename performed by mv is atomic provided that the file is moved within the same filesystem. The atomicity does not guarantee that the file is only in one place or another; it is quite possible that the file could be present in the filesystem in both places simultaneously for "a short time". What atomicity does guarantee, when offered, is that the file contents are instantaneously available completely and not partially. You can imagine that mv in such situations could have been implemented with ln followed by rm.

mv is most definitely not atomic when the move that it performs is from one filesystem to another, or when a remote filesystem cannot implement the mv operation locally. In these instances mv could be said to be implemented by the equivalent of a cp followed by rm.

Now, moving on to the question of atomicity across multiple files. mv is at best atomic only per file, so if you have a number of files to move together, the implementation is such that they will be moved one at a time. If you like, mv file1 dir; mv file2 dir; mv file3 dir.

If you really need a group of files to appear in a destination simultaneously, consider putting them in a directory and moving that directory. This single object (the directory) can be moved atomically.

edited Jan 24 '20 at 11:06

answered Mar 17 '15 at 20:00

roaima

1,567
13
26

4

'The atomicity does not guarantee that the file is only in one place or another; it is quite possible that the file could be present in the filesystem in both places simultaneously for "a short time".' -- Then what *does* the atomicity guarantee? That doesn't sound atomic at all. – John Kugelman Mar 17 '15 at 20:22
12

@JohnKugelman atomicity guarantees that the file is not partially in one place or the other; it's either complete or it doesn't exist in the filesystem. See [POSIX mv](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/mv.html) and [POSIX rename](http://pubs.opengroup.org/onlinepubs/9699919799/functions/rename.html) for the precise detail – roaima Mar 17 '15 at 20:31
Regarding single files, 1) Linux guarantees (man 2 rename), that if rename replaces an existing file, that "there is no point at which another process attempting to access newpath will find it missing". 2) POSIX [mentiones](http://pubs.opengroup.org/onlinepubs/9699919799/functions/rename.html) atomicity only in the "informative" section "Rationale" - and not in an unambiguous way. I wasn't able to find anything else regarding atomicity of rename(2). – Zrin Oct 27 '16 at 15:04
@Zrin rename(2) is only a part of `mv`. The question relates to `mv` so that is what I've addressed. You're correct in saying that a rename(2) call that replaces an existing file will ensure that the target filename isn't ever missing, but I'm not sure that it's relevant to the question as asked. – roaima Oct 27 '16 at 16:45
@roaima my point is that rename(2) is the only part of mv that is (partially) atomic... – Zrin Nov 07 '16 at 07:05

score 7 · Answer 2 · answered Mar 17 '15 at 19:25

7

No. mv dir1/* is the same as mv dir1/file1 && mv dir1/file2 && mv dir1/fileN. Each individual move is atomic, but not the full set.

answered Mar 17 '15 at 19:25

ceejayoz

32,469
7
81
105

5

The explanation is not correct, but the conclusion that each individual move is atomic is. The command is the same as `mv dir1/file1 dir1/file2 dir1/file3....`. – Thorbjørn Ravn Andersen Mar 17 '15 at 21:49
Doesn't `&&` means the first command must succeed in order for the second command to take place? So that would mean that if the first `mv` fails, it would stop. I think `mv` keeps attempting to move files. – Willem Van Onsem Mar 18 '15 at 01:46
@ThorbjørnRavnAndersen the underlying premise that `mv` is atomic is not necessarily true, and therefore the conclusion is also not always true. – roaima Feb 22 '17 at 18:20
@roaima Please feel free to elaborate. Perhaps even write a better answer. – Thorbjørn Ravn Andersen Feb 22 '17 at 20:07
@ThorbjørnRavnAndersen I have [already done so](http://serverfault.com/a/676231/267016). – roaima Feb 22 '17 at 21:08
@roaima Apparently this depends quite a lot on what "atomic" means. – Thorbjørn Ravn Andersen Feb 23 '17 at 08:30

score 1 · Answer 3 · answered Mar 18 '15 at 11:03

1

Another case, a new file is added to dir1 after the mv has started.

As the “*” is expanded by the shell, mv will not even know about the new file.

answered Mar 18 '15 at 11:03

Ian Ringrose

870
1
6
12

Is mv with wildcard still atomic

3 Answers3