bash find list of archived files with wildcard in while loop

Question

I have a few thousand gzipped text files in different subdirectories and used a subset of these files as input for a project a few years ago. Back then I had an unzipped copy of the files I actually used in one directory, but deleted this and kept only a list of these unzipped files in that folder

This was my initial idea, LIST is the list of files. PARENTDIR is the toplevel directory in which all files reside in various sub directories. The idea was to find all the archives in whatever sub directory they are and gunzip them to NEWDIR

#!/usr/bin/env bash    

LIST="listfile.txt"
PARENTDIR="/home/user/old/project"
NEWDIR="/home/user/old/project/2016"

while read line;
do
    ARCHIVE="$(find $PARENTDIR -name "$line*")"
    gunzip --stdout $ARCHIVE >$NEWDIR/$line
done <$LIST

I don't seem to get the find command right. It works without the variables, but not with, even without the command substitution, calling on the command line. My combination of quotes and wild cards is not quite correct, but I can't get it right, variable expansion doesn't help either and I guess I'm stuck...

Answer 1

I thought of using the -exec option for find but this would not work due to the redirection operator used with the gunzip command. One solution would be to perform the operation in two steps:

1. Copy the archives into $NEWDIR:

    while read line
    do
        find "$PARENTDIR" -name "$line*" -exec cp -v {} "$NEWDIR" \;
    done < "$LIST"

This should work with POSIX-compatible versions of find – not only GNU find.

Avoid over-writing of similarly named files

If you have duplicate filenames, they’ll be over-written in $NEWDIR. If you want to avoid over-writing the files, you’d have to recreate the directory tree inside $NEWDIR. This can be done using the install command from GNU coreutils which creates all parts of the path similar to mkdir -p.

    cd "$NEWDIR"
    while read line
    do
        find . -name "$line*" -exec install -D {} $NEWDIR/{} \;
    done < "$LIST"
    cd -

2. Decompress the copied files:

    find "$NEWDIR" -exec gunzip {} \;