Gzip huge directory into separate .gz files for ssh transfer

Question

I have a directory of ~200.000 .npy files with a total size ~100 gb. All files are stored directly below the main directory (i.e. there's no sub-directories). I need to transfer the directory and would like to do it by first compressing it into a smaller number of gzip files that I then transfer using ssh. I've tried to naïvely gzip the whole directory at once, which made my server freeze, requiring a hard reboot.

How can I easily gzip the directory of files into, say, 1000 .gz files that I can then easily transfer and unzip again?

I'd preferably like to do this in a manner, where the maximum resource consumption on the server at any one point (primarily RAM/IO) is agnostic regarding the characteristics of the directory (total size / # of files). I'm hoping that I can find a method that I'll be able to use with even larger directories without making my server freeze. The solution should preferably be using bash or Python. Thanks!

Answer 1

This is appears to be a good match for rsync. It will transparently compress the contents, and it can be told to limit the bandwidth usage, which serves both to avoid clogging the network, and to prevent high IO load on the originating server:

rsync -az --bwlimit=1m directory server:/destination/

-a tells rsync to copy the file metadata such as creation times, -z means use compression, and --bwlimit limits the bandwidth used over the network.

As an additional bonus when rsync is used, if you interrupt the operation for any reason, and re-run it again, it will automatically pick up where it left off. If you also need to delete extra files at the destination, add the --delete option.