27

I have a home file server running FreeNAS 8. A few days ago I used rsync to upload my entire iTunes library from Mac so that I could load my library over the network instead of off a slow USB drive. This mostly worked, and iTunes runs much better now, but I'm running into issues accessing any songs that have non-ascii characters in it (I first noticed the problem when loading Queensrÿche tracks). The files would show up in the Finder, but any attempt to access them made them vanish until I reconnected to the server.

After some research I found out this is because OSX uses a different UTF character order from Linux. OSX filesystems use Unicode Normalization Form D (NFD), where linux uses Form C (NFC). Rsync doesn't convert these forms when it performs the copy from my mac to the server, now when iTunes tries to access a file with a special character over the network, the files on the server have the wrong encoding and afpd reports they don't exist.

What is the best way to address this problem? Is it possible to make rsync perform the unicode conversion while uploading the base library to the server? Can I configure afpd to transmit/receive filenames in NFD format? Is there an easy solution to change the filenames on the server? I found some stuff about a program named convmv, but I don't know if I can run that on FreeNAS.

Twipped
  • 613
  • 2
  • 7
  • 10
  • 1
    Sounds like a bug with the OS X version of rsync. – Ignacio Vazquez-Abrams Jun 10 '12 at 23:44
  • I know this is a real old thread but would like to add comments on my recent (Catalina) experience regarding this. It appears to me that Apple's new APFS file system is NFD/NFC agnostic. i.e. the files system accepts both for filenames (unlike HFS+ which forces NFD). My Synology NAS also accepts both NFC and NFD encoded files. Futhermore, The Mac iTunes app writes files with NFC encoding. So I've decided to convert my data on my APFS disk to NFC. rsync (via ssh) works great without any --iconv parameter. However, It appears that when mounting the NAS file system (either SMB or AFP) all the fil – Mark Lederer Feb 05 '20 at 22:07

5 Answers5

57

You can use rsync's --iconv option to convert between UTF-8 NFC & NFD, at least if you're on a Mac. There is a special utf-8-mac character set that stands for UTF-8 NFD. So to copy files from your Mac to your NAS, you'd need to run something like:

rsync -a --iconv=utf-8-mac,utf-8 localdir/ mynas:remotedir/

This will convert all the local filenames from UTF-8 NFD to UTF-8 NFC on the remote server. The files' contents won't be affected.

Daniel Serodio
  • 249
  • 3
  • 10
Lennart L
  • 671
  • 5
  • 3
  • 2
    I'm not the Original Poster so it's not up to me, but this is a much much cleaner and efficient solution than the one flagged as accepted. Starring this for sure, will be so useful. – ItsGC Sep 13 '12 at 21:20
  • 1
    Great answer; I had no idea that `UTF8-MAC` stands for NFD; when used with `iconv` itself, this provides a generic mechanism for translating back and forth between NFC and NFD. – mklement May 07 '14 at 15:00
  • great answer, this solves a long standing problem in sync a mac with a linux server! – meduz Nov 01 '15 at 12:53
  • 2
    On mac you may also need `brew tap homebrew/dupes && brew install homebrew/dupes/rsync && rehash` – SaveTheRbtz Apr 20 '16 at 06:05
  • 1
    I get `rsync: --iconv: unknown option` – KMC Dec 26 '18 at 20:57
  • @KMC You need to have at least version 3.0.0 of rsync. Mac OS used to come with very dated versions of many common tools... – LCC Feb 26 '20 at 11:43
7

Currently I'm using rsync --iconv like this:

Copying files from Linux server to OS X machine

You should execute this command from OS X machine:

rsync -a --delete --iconv=UTF-8-MAC,UTF-8 'username@server.ip.address.here:/home/username/path/on/server/' /Users/username/path/on/machine/

Copying files from OS X machine to Linux server

You should execute this command from OS X machine:

rsync -a --delete --iconv=UTF-8-MAC,UTF-8 /Users/username/path/on/machine/ 'username@server.ip.address.here:/home/username/path/on/server/'
Envek
  • 223
  • 3
  • 7
4

Note: If you are using version 3.0.0 or newer of rsync, the --iconv option as mentioned in the other answers is clearly the superior solution.

Something that should work is rsyncing between the source directory and the mounted remote file system (SMB, NFS, AFP), which rsync will just treat as local file system.

However, I do not know how well this works in practice, and you have to work around different issues, for example the delta-transfer algorithm won’t be used by default (since source and destination are “local”) (maybe --no-whole-file will work?), you have to check,e.g., that SMB effectively preserves modification times, etc.

LCC
  • 167
  • 7
  • This is ultimately what I ended up doing. I deleted the entire collection from the NAS and ran rsync again, using the locally mounted CIFS connection instead of the rsync daemon on the NAS. Now I'm just fixing itunes issues from filename capitalization. :/ – Twipped Jul 15 '12 at 23:52
1

Don't use rsync to copy the files to your NAS. When you use rsync to copy the files the filenames will be stored on your NAS in UTF NFD format (i.e. the OSX format) but Samba server running on your NAS only understands UTF NFC format filenames. Use the CIFS/SMB (Samba) interface to copy the files and the everything will be fine.

whyme
  • 11
  • 1
0

From my experience I recommend using SMB instead of ssh. Iconv solves problem with encoding, but there is still problem with allowed characters on diffrent systems:

Original file name on Mac:

https-//img-9gag-fun.9cache.com/photo/adK9jzN_460s

After copy by rsync over SMB:

-as seen by Mac (over SMB):     https-//img-9gag-fun.9cache.com/photo/adK9jzN_460s
-as seen by Ubuntu (over SMB):  https-img-9gag-fun.9cache.comphotoadK9jzN_460s
-as seen by Windows10 (over SMB):   https-∀∀img-9gag-fun.9cache.com∀photo∀adK9jzN_460s
-as seen by Ubuntu server locally:  https-img-9gag-fun.9cache.comphotoadK9jzN_460s

After copy by rsync over ssh (with ant without iconv flag):

-as seen by Mac (over SMB):     H0INHQ~6
-as seen by Ubuntu (over SMB):  H0INHQ~6
-as seen by Windows10 (over SMB):   H0INHQ~6
-as seen by Ubuntu server locally:  https-::img-9gag-fun.9cache.com:photo:adK9jzN_460s
lukdz
  • 1