Remote copy of LMDB

Question

I want to migrate an LMDB from my local machine to another remote machine, but there is some weirdness about the file size. According to the filesystem, an LMDB is a directory containing two files: data.mdb and lock.mdb.

The output of ls -altoh lmdb indicates that data.mdb has a file size of 4T, which matches the map_size parameter I used to create the LMDB. All this means is that when the DB is opened, the OS will memory map the file, giving it 4T of virtual space. The output of du -hs lmdb indicates that the lmdb is taking up ~900MB of disk, which agrees with the map_size reported by python -mlmdb -e lmdb stat.

When I do a local copy cp -r lmdb lmdb_copy, it works as expected: 900MB of data is copied. The same when I do scp -r lmdb lmdb_copy2 (using scp to do a local copy).

However, when I do a remote copy scp -r lmdb user@remotehost:~/lmdb_copy, scp attempts to copy 4T of data, as indicated by the progress bar. I stopped the scp after 2GB of data has been transfered.

On the remote machine, 'ls and du both 2GB as the size of the LMDB. python -mlmdb -e lmdb_copy stat reports the correct size of 900MB and that all of the entries are there. I've verified that I can print out all of the keys and they are correct.

With this background, my question is, why does scp attempt to copy all 4T of the memory map size? Ideally, I'd like to let scp do its thing in the background without having to manually kill it.

Answer 1

You could try using rsync to do the copy. It says it deals with sparse files. Something like

rsync --rsh=ssh --archive --sparse lmdb user@remotehost:~/lmdb_copy

As an aside, and some insight into why scp works locally but not over a network, when scp sees that it's a local to local copy it just passes the request to the cp command directly. Monitoring an scp command's system calls, I caught it doing this

execve("/bin/sh", ["sh", "-c", "exec cp -r foo bah"], [/* 20 vars */])