Change line endings and encoding of file at the same time

Question

I have some files with Windows line endings and latin-1 encoding that I need to convert to unix line endings and utf-8.

Of course I can

for file in ./*.csv; do
    sed s'/^M//' "$file" > "${file}.bak"
    iconv -f iso-8859-1 -t utf8 "${file}.bak" > "$file"
    rm "${file}.bak"
done

But is there a commonly available tool that can do both of these things at once? Maybe this isn't the most efficient way. (Maybe iconv?)

Answer 1

I would make some slight modifications to your script. First don't use ls in your for loop, use *.csv because the glob will handle non-printable characters and spaces in file names. Instead of using sed's inplace redirect to $file.bak. If strings is available on your system then replace sed with strings. And always remember to quote variables.

    for file in *.csv
    do    sed 's/^M//' "$file" > "${file}.bak"
         #strings "$file" > "${file}.bak"
          iconv -f -iso-8859-1 -t utf8 "${file}.bak" > "$file"
          rm "${file}.bak"
    done