Weird bug in 'tar' not including files named .__init__.py

Question

Does anyone know why tar is not including files named .__init__.py (note the dot)?

$ mkdir /tmp/work && cd /tmp/work
$ mkdir foo
$ touch foo/.__init__.py
$ touch foo/.namespace__init__.py
$ tar czf foo.tar.gz foo

$ mkdir e && mv foo.tar.gz e/ && cd e/
$ tar zxf foo.tar.gz
$ ls -al foo/
total 0
drwxr-xr-x  2 sridharr  wheel  102 14 Mar 17:16 .
drwxr-xr-x  3 sridharr  wheel  136 14 Mar 17:17 ..
-rw-r--r--  1 sridharr  wheel    0 14 Mar 17:16 .namespace__init__.py
$ 

$ echo ".__init__.py file is missing. WTF? This is OSX 10.6"

Update: tar appears to be ignoring files starting with ._ characters; why?

Update 2: I cannot reproduce this on Linux.

Answer 1

Found the solution. On Apple systems, files starting with ._ contains resource fork of another file. They are not normal files that can be copied and moved around.

Answer 2

There are some undocumented(?) environment variables that can be used to disable the special handling of extended attributes and/or resource forks in tar (and pax, for what it is worth). rsync has the -E/--extended-attributes option to enable(!) this handling—but on some non-Apple rsyncs -E means --executability instead.

On Mac OS X 10.4 (the first release that created these AppleDouble-encoded ._* archive members), the environment variable is COPY_EXTENDED_ATTRIBUTES_DISABLE. In Leopard and Snow Leopard, the variable is COPYFILE_DISABLE. Typically the variables just have to be set. Any value will do (even the empty string), but true seems to be traditional. Thus:

COPY_EXTENDED_ATTRIBUTES_DISABLE=true COPYFILE_DISABLE=true tar …

Setting this variable has the following effects:

• When creating/updating archives:
  • Prevents the creation of ._* archive members when archiving files with extended attributes.
  • Allows the creation of ._* archive members when archiving actual ._* files.
• When extracting archives:
  • Causes ._* archive members to be extracted as plain files instead of restoring the extended attributes to the related file.

In short, setting these variables makes tar, et al. act like they would on (e.g.) Linux.

If you rarely need to archive files that have extended attributes or resource forks, and you might need to archive or extract actual ._* files, then you might consider setting and exporting these variables in one of your shell initialization files:

# Tell tar, pax, etc. on Mac OS X 10.4+ not to archive
# extended attributes (e.g. resource forks) to ._* archive members.
# Also allows archiving and extracting actual ._* files.
COPY_EXTENDED_ATTRIBUTES_DISABLE=true COPYFILE_DISABLE=true
export COPY_EXTENDED_ATTRIBUTES_DISABLE COPYFILE_DISABLE

---

These ._* files are also used to store extended attributes on filesystems that do not support them—most commonly the FAT variants. These variables will not really help when dealing with ._* files on other filesystems, just archives.

The HFS+ filesystem used in Mac OS X is perfectly capable of storing actual ._* files, so once you use the variables to extract the files to the filesystem, the files can be accessed properly in all the normal ways.

Answer 3

I cannot replicate this on a Debian 5.0 host. Perhaps there is a bug in the version of tar installed on the system you are using? What version of *nix are you using?

$ mkdir foo
$ touch foo/.namespace__init__.py
$ touch foo/.__init__.py
$ tar -czvf foo.tar.gz foo/
foo/
foo/.namespace__init__.py
foo/.__init__.py

$ # example the file
$ tar -tzvf foo.tar.gz
drwxr-xr-x cfrancy/cfrancy   0 2010-03-14 17:34 foo/
-rw-r--r-- cfrancy/cfrancy   0 2010-03-14 17:34 foo/.namespace__init__.py
-rw-r--r-- cfrancy/cfrancy   0 2010-03-14 17:34 foo/.__init__.py