Generate a files list and truncate it in 2 list of X Mb of files each

Question

I need to generate a list of all files in a certain dir, and truncate it in X lists containing each one a fixed amount of data of files.

E.g. I have 95 Gb of data.

• generate the file list (total.txt)
• split total.txt in 3 lists:
  1. slice1.txt containing the list of the 1st 35 Gb of files
  2. slice2.txt containing the list of the following 35 Gb of files
  3. slice3.txt containing the list of the remaining files

Any hint? I've googled and played around with find, awk, grep, but this task seems really above my competencies.

Answer 1

You can try using the split command to separate your files:

split total.txt -b 4444160

The above would split total.txt into 35 GB files.

Answer 2

while read filename; do cat $filename; done < total.txt | split -b 35G - slice

This will create "sliceaa", "sliceab", "sliceac" which you can rename.

With other implementation of split, you may have to say -b 35000m

If you have bash, you can write

cat $(< total.txt) | split -b 35G - slice

assume there are not hundreds or thousands of filenames.

To create total.txt

files=(*)
printf "%s\n" "${files[@]}" > total.txt