How to zip lots of folders into three+ independent zip files?

Question

I have a massive amount of folders, and I want to zip/gzip/(what ever) them into 3 or more independent files.

So for example, let's say I have this directories on my server:

/dir1
/dir2
/dir3
/dir4
/dir5
/dir6

I need to somehow zip them in 3 separate zip files. That will look like this:

collection1.zip

/dir1
/dir2
/dir3

collection2.zip

/dir4
/dir5
/dir6

collection3.zip

/dir7
/dir8
/dir9

You can use `rsync` over `ssh`, no need to zip, no need to delete anything. — dawud, Jul 09 '14 at 10:38
Target server support only FTP access, no SFTP and my server doesn't have FTP command. So can you maybe give me example of how I would transfer public_html to another server by using rsync? — Jinx, Jul 09 '14 at 10:48
@dawud i have tried rsync but it uses sftp and target server doesn't support sftp nor shell access. — Jinx, Jul 09 '14 at 10:55
Maybe you can use `ncftp` or `lftp`. Take a look at [this Q/A](http://serverfault.com/questions/24622/how-to-use-rsync-over-ftp). — dawud, Jul 09 '14 at 10:58
@dawud Obviously not, otherwise I'd up the storage limit already. I just have access to accounts cPanel, not WHM. — Jinx, Jul 09 '14 at 12:30
If disk I/O and CPU usage are not a concern: You could loop through all files under a folder and add to a zip (one at a time) until the zip reaches a set size.. change the zip name and continue filling up a new zip... — Lars, Jul 09 '14 at 12:56

joeg1ff · Answer 1 · 2014-07-09T16:11:44.783

2

Sorry I dont have a new enough cred to just comment. But something like this should help you with the zipping of the folders.

#! /bin/bash
DIR=$1
i=0
arr=()
zipnum=0
cd $DIR
for item in *
do
    if [[ -d $item ]]; then
            if [ $i -lt 2 ]; then 
                    arr+=($item)
                    ((i++))
            else
                    arr+=($item)
                    zip ${arr[@]} $zipnum
                    ((zipnum++))
                    i=0
                    arr=()
            fi
    fi
done
((zipnum++))
zip ${arr[@]} $zipnum

It should work (didn't have time to test it) Save the file as zip.sh and run is as ./zip.sh /your/directory/here As for the file transfer couldn't you just install an ftp client on the server and ftp the files?

edited Jul 09 '14 at 16:11

answered Jul 09 '14 at 15:56

joeg1ff

391
3
6

Didn't test it either, but I have found my own solution. Thanks for the effort anyway. – Jinx Jul 09 '14 at 16:30
This created a zip of folders for me, no files. – Lars Jul 09 '14 at 17:32
@Lars judging by his post he only wanted to zip up the folders. If you want files as well just remove the if [[ -d $item ]] statement. – joeg1ff Jul 09 '14 at 20:17

score 1 · Answer 2 · answered Jul 09 '14 at 17:31

The below script was tested on CentOS 6.5.

Walks through the files under a specified folder and adds them one by one to zip file(s) no bigger than a specified max-size. Adding files to a zip one at a time can cause CPU or Storage I/O spikes:

#!/bin/bash

files_folder="/home/username/public_html/"
zip_folder="/home/username/"
zip_maxsize=524288000  # 500 MB

zip_prefix="archive"
zip_suffix=0

cd ${files_folder}

for FILE in $(find . -type f); do
    zip_file="${zip_folder}${zip_prefix}-${zip_suffix}.zip"
    if [ -f ${zip_file} ]; then
        zip_size=$(ls -l ${zip_file} |cut -d' ' -f5)
    fi
    file_size=$(ls -l ${FILE} |cut -d' ' -f5)
    comb_size=$((zip_size + file_size))
    if [ ${comb_size} -gt ${zip_maxsize} ]; then
        zip_suffix=$((zip_suffix + 1))
        zip_file="${zip_folder}${zip_prefix}-${zip_suffix}.zip"
    fi
    zip -q ${zip_file} ${FILE}
done

score 0 · Accepted Answer · answered Jul 09 '14 at 16:28

I have found a simple solution.

First I made a list of all folders I need to zip:

ls > list.txt

Than I removed ./ and ../ from the list.

After that I have split that list with text editor into 5 different files, e.g. listPart1, listPart2, listPart3 ...

I put those lists in the folder that contains all folders that need to be zipped and I ran this command from that folder for each list:

cat listPart1 | zip -r@ part1.zip

How to zip lots of folders into three+ independent zip files?

3 Answers3