performance when appending to a .7z archive

0

i have several large (mostly text) files that i want to compress

files are stored in dumps in subdirectories named after the date the dump was taken

some of these files are single file archives (none using tar) in their own compressed format, particularly bzip2 and gz, and sometimes 7z

the extracted files are always the same filename sans the compression extension (eg. filename.xml.gz -> filename.xml)

i want to write a script that creates a .7z archive spanned across 2gb files, and update this archive every 10-30 days (or when a new 8 digit directory appears in a specific directory on an ftp site) after using rsync on this remote directory

then i want to extract all .7z, .bz2, and .gz files, and if successful, delete the original compressed files

before creating the spanned .7z files, i want this script to create a local php server based on the .xml and .sql files and then run wget recursively on the result, then delete the .xml and .sql files if successful

then, finally, i want to create the .7z spanned archive based on the .html files that wget creates, then delete the contents of the php server

i want to do this with as much data verification as possible, if not a byte-by-byte comparison, then at least 2 different checksum verifications

once i have this script in place, i want to append to the spanned .7z archive

is performance a significant issue with appending with 7zip? in my research i've found zip archives are terrible for appending, but i found nothing with 7zip performance in particular

should i extract the contents of the spanned .7z, then create a new archive? or can i just append to the archive without any significant performance (time) issues?

also, any guidance on writing this script would be greatly appreciated

loud_flash

Posted 2019-11-03T11:05:00.317

Reputation: 1

No answers