cp is too slow, same with rsync (I assume) - is there a way to share files in two locations?

Question

I am using Docker and Docker cannot COPY symlinked files into the image. But the files that are symlinked are not in the 'build context'. So I was going to copy them into the build context with cp, but that's really slow. Is there some way to share the files on two different locations on disk without have to copy them and without using symlinks?

Answer 1

You could copy using reflinks `cp --reflink=always', which will create a differential file that references a static original. If you do this, you'll probably want to make the original immutable so the reflinked copy doesn't get borked.

Or you could use something like LVM snapshots to quickly create / clone filesystems that are differentials of an original. Thin LVM would be best for this approach, and you would accomplish this by mounting the snapshot itself rather than using a snapshot as a way to roll back an origin volume like we traditionally do. Using LVM would be safer than reflinked copies, as changes to the original are handled gracefully by updating all snapshots with origin updates.

Third, you could snapshot BTRFS subvolumes. This is the best of both worlds, as it's a filesystem or file level snapshot mechanism that doesn't require you to manage both files an a volume manager separately. This process would be very similar to LVM, as you can mount subvolumes wherever you please.

And finally, have you considered NFS? Specifying an NFS connection in a container image is a good way to get shared storage.