Linux copy to fat32 filesystem: invalid argument

Question

When I copy files from an ext3 partition to a fat32 one using cp:

cp -R /ext3/stuff /fat32/partition/

I get invalid argument messages for all files with colons and question marks in.

Is there any way to get cp to strip out the invalid characters for the target filesystem?

edit: I've checked through cp's options again, and unless I'm being stupid, there's nothing in there. I'm sure I could write a script, but it feels like there should be a cleaner solution!

Answer 1

The usual suspects when you want complex copies or renames are GNU cp, zmv from zsh, rsync and pax (or cpio). There's no rename feature in cp, nor (I think) in rsync. While zmv can rename, this doesn't mesh well with recursive copies. But pax can do it:

cd /ext3
pax -rw -s '/[*?:]/_/gp' stuff /fat32/partition

This changes each *?: to _. Warning: minimally tested. If there are collisions, whichever file is copied last wins.

Answer 2

Based on post by Gilles I tested the following list:

#!/bin/sh
touch questionmark?
touch less<
touch less\<
touch more\>
touch backslash\\
touch colon:
touch asterisk\*
touch pipe\|
touch inch\"
touch carret\^
touch comma,
touch semicolon\;
touch plus+
touch equals=
touch lbracket[
touch rbracket]
touch quote\'

I tried to copy that onto Android phone MicroSDHC card with vfat filesystem and refined pax command until everything worked. That may still not be enough for Windows and Unicode:

pax -rw -s '/[?<>\\:*|\"]/_/gp' source dest

You may also want to use the -k option to ensure that there are no overwrites (due to collisions in file names). Both lists I gave in the comment were different from Linux vfat behaviour.

Answer 3

After reading the answers to this very interesting question and doing some experimenting with SD card for Android (exfat) and a car entertainment system (vfat), I came up whis this little bash script.

#! /bin/bash

DST=$1
# copy music to FAT media

find music/ Music/ -type f | while read f ; do
    d=$DST/$( echo $f | sed 's/[^-A-Za-z0-9/._ ()]/_/g' )

    echo :$d:
    mkdir -p "$(dirname "$d")"
    cp -n "$f" "$d"

done

It take the destination (mount point) as it argument and use find to locate all files in my music repository.

For each file path, it computes a destination path and file name by prepending the destination and replacing any offending character with an underscore _. I use a white list of characters (letters, digits, -, /, ., . (, ) and _) to remove any unwanted punctuation.

Depending on the locale, this will leave accented letters in the path and file name, which is OK for modern FAT file systems, as it seems.

For each destination file path and name, the directories are created as needed using mkdir -p, then the file is copied, unless it already existed.

Note the quote " chars in various places, they are required to keep paths and names with spaces in them from breaking apart.

Answer 4

I received the "Invalid argument" when copying with cp -r source usbstick and found out the cause was a sourcefilename ending with a space. Removing the space cleared the message. The file with incorrect name happened BTW in this case to be in mailer program directories.

Answer 5

I just had cause to need to do this, and while the pax-based answer was good, it still ran into problems with accented characters.

So I found it simpler to use tar and get it to replace all the non-permitted characters with underscores:

cd /parent-of-source
tar cf - Söurce | (cd /destination; tar xvf - --transform='s/[^A-Za-z0-9\/ ]/_/g')

It is doubtless possible to come up with a better list of permitted characters than the one above, but this works.