This is my effort of trying to make a deterministic tarball from an SVN repo.
Requires GNU tar 1.27 or later.
Features:
- Deterministic tarball
- Timestamp precision in microseconds from SVN, thanks to
pax
extended header
- Revision ID stored in archive comments just like
git archive
would do
- I demonstrate both
.tar.gz
and .tar.xz
compressions. For gzip, you may optimize the compression further by using advdef
from AdvanceCOMP (advdef
uses zopfli library).
As an example, I use subversion repo itself as a source for checkout and
creation of the tarball. Note that this isn't the way SVN package their
tarballs and I'm no way related to SVN development. It's just an example,
after all.
# URL of repository to export
url="https://svn.apache.org/repos/asf/subversion/tags/1.9.7/"
# Name of distribution sub-directory
dist_name=subversion-1.9.7-test
# ---------------------------------------------------------------------
info=$(svn info --xml "$url" | tr -- '\t\n' ' ')
revision=$(echo "$info" |
sed 's|.*<commit[^>]* revision="\{0,1\}\([^">]*\)"\{0,1\}>.*|\1|')
tar_name=${dist_name}-r${revision}
# Subversion's commit timestamps can be as precise as 0.000001 seconds,
# but sub-second precision is only available through --xml output
# format.
date=$(echo "$info" |
sed 's|.*<commit[^>]*>.*<date>\([^<]*\)</date>.*</commit>.*|\1|')
# Factors that would make tarball non-deterministic include:
# - umask
# - Ordering of file names
# - Timestamps of directories ("svn export" doesn't update them)
# - User and group names and IDs
# - Format of tar (gnu, ustar or pax)
# - For pax format, the name and contents of extended header blocks
umask u=rwx,go=rx
svn export -r "$revision" "$url" "$dist_name"
# "svn export" will update file modification time to latest time of
# commit that modifies the file, but won't do so on directories.
find . -type d | xargs touch -c -m -d "$date" --
trap 's=$?; rm -f "${tar_name}.tar" || : ; exit $s' 1 2 3 15
# pax extended header allows sub-second precision on modification time.
# The default extended header name pattern ("%d/PaxHeaders.%p/%f")
# would contain a process ID that makes tarball non-deterministic.
# "git archive" would store a commit ID in pax global header (named
# "pax_global_header"). We can do similar.
# GNU tar (<=1.30) has a bug that it rejects globexthdr.mtime that
# contains fraction of seconds.
pax_options=$(printf '%s%s%s%s%s%s' \
"globexthdr.name=pax_global_header," \
"globexthdr.mtime={$(echo ${date}|sed -e 's/\.[0-9]*Z/Z/g')}," \
"comment=${revision}," \
"exthdr.name=%d/PaxHeaders/%f," \
"delete=atime," \
"delete=ctime")
find "$dist_name" \
\( -type d -exec printf '%s/\n' '{}' \; \) -o -print |
LC_ALL=C sort |
tar -c --no-recursion --format=pax --owner=root:0 --group=root:0 \
--pax-option="$pax_options" -f "${tar_name}.tar" -T -
# Compression (gzip/xz) can add additional non-deterministic factors.
# xz format does not store file name or timestamp...
trap 's=$?; rm -f "${tar_name}.tar.xz" || : ; exit $s' 1 2 3 15
xz -9e -k "${tar_name}.tar"
# ...but for gzip, you need either --no-name option or feed the input
# from stdin. This example uses former, and also tries advdef to
# optimize compression if available.
trap 's=$?; rm -f "${tar_name}.tar.gz" || : ; exit $s' 1 2 3 15
gzip --no-name -9 -k "${tar_name}.tar" &&
{ advdef -4 -z "${tar_name}.tar.gz" || : ; }
I don't know about svn, but github allows you to download .tar.gzs of a git repo. – Macha – 2009-11-29T13:54:28.750