Is there a way to create a copy-on-write copy of a directory?

Question

I'm thinking of a situation where I would have something that creates a copy of a directory, tweaks a few files, and then does some processing on the result. This wold be done fairly often, maybe a few dozen times a day. (The exact use case is testing patch submissions; dupe the code, patch it, build/test/report/etc.)

What I'm looking for could be done by creating a new directory structure and populating it with hard links from the origonal. However this only works if all the tools you use delete and recreate files rather than edit them in place.

Is there a way to have the file system do copy-on-write for a file?

Note: I'm aware that many FSs use COW at a block level (all updates are done via writes to new blocks) but this is not what I want.

It doesn't appear maintained, but perhaps http://boklm.eu/copyfs/ — Zoredache, Apr 07 '10 at 03:07
So you want to do copy-on-write via the filesystem, but you don't want a filesystem that uses copy-on-write? I'm confused. — Kamil Kisiel, Apr 07 '10 at 05:02
@Kamil Kisiel: I want file-level COW not block-level COW. File-level COW results in `cp` and `ln` doing almost exactly the same thing. Block-level COW is a trick to minimize the interval that the FS meta data is incorrect: http://en.wikipedia.org/wiki/ZFS#Copy-on-write_transactional_model — BCS, Apr 07 '10 at 15:01
It's speculative enough that unless you know of a way to do it for some setup (in that case just post it) I wouldn't want people burning time figuring out how to do it (unless they are doing it for fun :) — BCS, Apr 08 '10 at 15:19

score 8 · Answer 1 · edited Oct 19 '18 at 03:28

8

If you have your choice of platform for your fileserver, I'd go with a recent OpenSolaris build and use the deduplication feature of ZFS. This way copies of files would take up no additional space, and even common segments between files would not be replicated.

You can read all about ZFS deduplication at this post on Jeff Bonwick's Blog.

edited Oct 19 '18 at 03:28

Solomon Ucko

107
4

answered Apr 07 '10 at 05:03

Kamil Kisiel

11,946
7
46
68

1

ZFS is amazing; it will change the way you think about storage – tegbains Apr 07 '10 at 05:39
Neat, unless there's a down side to having it on for the whole drive. – BCS Apr 07 '10 at 14:54
I added a link to a blog post that has more technical details. You can see if it's what you need there. – Kamil Kisiel Apr 08 '10 at 03:57
New url for post of Jeff Bonwick: https://blogs.oracle.com/bonwick/zfs-deduplication-v2 – Mathieu Jun 27 '18 at 09:06

score 0 · Answer 2 · answered May 28 '20 at 21:20

I find this question while searching a better solution instead of my current, which is like: cp -al SOURCEDIRNAME copy.SOURCEDIRNAME or

find SOURCEDIRNAME -type d -exec mkdir -p copy.{} ';'
find SOURCEDIRNAME -type f -exec ln {} copy.{} ';'

At this point i have a copy of the structure with a bunch of hardlinks. This works relative fast, and don't need one time more disk space, cause we do not rly copy all files. Now i can do some ''stuff'' (like patch or build) on the copy directory. If they do not modify the original files, only rename them and/or create new files. After the work, i have some results and can remove the copy directory and start over.

If you use more complicated tools and dont trust them (or you know they try to overwrite existing files) you may need cowdancer.

score 0 · Answer 3 · answered May 28 '20 at 23:54

Try use one of UniouFS, AUFS or Overlay Filesystem (depends on which supported in Your distro)

For example OverlayFS used by docker as one of drivers to merge layers

Overlay Filesystem

Overlaying mainly involves directories. If a given name appears in both upper and lower filesystems and refers to a non-directory in either, then the lower object is hidden - the name refers only to the upper object.

Where both upper and lower objects are directories, a merged directory is formed.

At mount time, the two directories given as mount options “lowerdir” and “upperdir” are combined into a merged directory:
mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,workdir=/work /merged
The “workdir” needs to be an empty directory on the same filesystem as upperdir.

ankostis · Answer 4 · 2019-03-07T17:38:58.993

0

Most linux virtualization solutions offer file-level COW among virtualized OSes (copied from the WP):

YES:
- Docker, Linux-VServer, lmctfy, LXC, Singularity, Virtuozzo, Sandboxie, systemd-nspawn, Solaris Containers(ZFS), FreeBSD jail(ZFS), OpenVZ
NO:
- chroot, sysjail, WPARs, iCore Virtual Accounts, Turbo

edited Mar 07 '19 at 17:38

answered Feb 08 '19 at 16:34

ankostis

101
3

While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes – Romeo Ninov Feb 08 '19 at 20:01

score -2 · Answer 5 · answered Apr 07 '10 at 05:56

-2

May be SVN or other versioning tool (git, cvs)?

answered Apr 07 '10 at 05:56

Dmitry Trukhanov

378
1
6

That's even worse because it creates two copies of each file and sucks them down over a socket. – BCS Apr 07 '10 at 14:52

Is there a way to create a copy-on-write copy of a directory?

5 Answers5