tar (computing)

In computing, tar is a computer software utility for collecting many files into one archive file, often referred to as a tarball, for distribution or backup purposes. The name is derived from "tape archive", as it was originally developed to write data to sequential I/O devices with no file system of their own. The archive data sets created by tar contain various file system parameters, such as name, timestamps, ownership, file access permissions, and directory organization. The command line utility was first introduced in the Version 7 Unix in January 1979, replacing the tp program.[2] The file structure to store this information was standardized in POSIX.1-1988[3] and later POSIX.1-2001,[4] and became a format supported by most modern file archiving systems.

For other uses, see Tar (disambiguation).

tar

Developer(s)	AT&T Bell Laboratories
Initial release	January 1979 (1979-01)
Stable release	1.32 / February 23, 2019 (2019-02-23)
Operating system	Unix and Unix-like, Microsoft Windows
Type	Command

tar

Filename extension	.tar
Internet media type	application/x-tar
Uniform Type Identifier (UTI)	public.tar-archive
Magic number	u s t a r \0 0 0  at byte offset 257 (for POSIX versions) u s t a r \040 \040 \0  (for old GNU tar format)[1] absent in pre-POSIX versions
Latest release	various (various)
Type of format	File archiver
Standard	POSIX since POSIX.1, presently in the definition of pax
Open format?	Yes

Rationale

Many historic tape drives read and write variable-length data blocks, leaving significant wasted space on the tape between blocks (for the tape to physically start and stop moving). Some tape drives (and raw disks) support only fixed-length data blocks. Also, when writing to any medium such as a filesystem or network, it takes less time to write one large block than many small blocks. Therefore, the tar command writes data in blocks of many 512 byte records. The user can specify a blocking factor, which is the number of records per block; the default is 20, producing 10 kibibytes(KiB) blocks.[5]

File format

A tar archive consists of a series of file objects, hence the popular term tarball, referencing how a tarball collects objects of all kinds that stick to its surface. Each file object includes any file data, and is preceded by a 512-byte header record. The file data is written unaltered except that its length is rounded up to a multiple of 512 bytes. The original tar implementation did not care about the contents of the padding bytes, and left the buffer data unaltered, but most modern tar implementations fill the extra space with zeros.[6] The end of an archive is marked by at least two consecutive zero-filled records. (The origin of tar's record size appears to be the 512-byte disk sectors used in the Version 7 Unix file system.) The final block of an archive is padded out to full length with zeros.

Header

The file header record contains metadata about a file. To ensure portability across different architectures with different byte orderings, the information in the header record is encoded in ASCII. Thus if all the files in an archive are ASCII text files, and have ASCII names, then the archive is essentially an ASCII text file (containing many NUL characters).

The fields defined by the original Unix tar format are listed in the table below. The link indicator/file type table includes some modern extensions. When a field is unused it is filled with NUL bytes. The header uses 257 bytes, then is padded with NUL bytes to make it fill a 512 byte record. There is no "magic number" in the header, for file identification.

Pre-POSIX.1-1988 (i.e. v7) tar header:

Field offset	Field size	Field
0	100	File name
100	8	File mode
108	8	Owner's numeric user ID
116	8	Group's numeric user ID
124	12	File size in bytes (octal base)
136	12	Last modification time in numeric Unix time format (octal)
148	8	Checksum for header record
156	1	Link indicator (file type)
157	100	Name of linked file

The pre-POSIX.1-1988 Link indicator field can have the following values:

Link indicator field

Value	Meaning
'0' or (ASCII NUL)	Normal file
'1'	Hard link
'2'	Symbolic link

Some pre-POSIX.1-1988 tar implementations indicated a directory by having a trailing slash (/) in the name.

Numeric values are encoded in octal numbers using ASCII digits, with leading zeroes. For historical reasons, a final NUL or space character should also be used. Thus although there are 12 bytes reserved for storing the file size, only 11 octal digits can be stored. This gives a maximum file size of 8 gigabytes on archived files. To overcome this limitation, in 2001 star introduced a base-256 coding that is indicated by setting the high-order bit of the leftmost byte of a numeric field. GNU-tar and BSD-tar followed this idea. Additionally, versions of tar from before the first POSIX standard from 1988 pad the values with spaces instead of zeroes.

The checksum is calculated by taking the sum of the unsigned byte values of the header record with the eight checksum bytes taken to be ascii spaces (decimal value 32). It is stored as a six digit octal number with leading zeroes followed by a NUL and then a space. Various implementations do not adhere to this format. For better compatibility, ignore leading and trailing whitespace, and take the first six digits. In addition, some historic tar implementations treated bytes as signed. Implementations typically calculate the checksum both ways, and treat it as good if either the signed or unsigned sum matches the included checksum.

Unix filesystems support multiple links (names) for the same file. If several such files appear in a tar archive, only the first one is archived as a normal file; the rest are archived as hard links, with the "name of linked file" field set to the first one's name. On extraction, such hard links should be recreated in the file system.

UStar format

Most modern tar programs read and write archives in the UStar (Unix Standard TAR[2][7]) format, introduced by the POSIX IEEE P1003.1 standard from 1988. It introduced additional header fields. Older tar programs will ignore the extra information (possibly extracting partially named files), while newer programs will test for the presence of the "ustar" string to determine if the new format is in use. The UStar format allows for longer file names and stores additional information about each file. The maximum filename size is 256, but it is split among a preceding path "filename prefix" and the filename itself, so can be much less.[8]

Field offset	Field size	Field
0	156	(several fields, same as in old format)
156	1	Type flag
157	100	(same field as in old format)
257	6	UStar indicator "ustar" then NUL
263	2	UStar version "00"
265	32	Owner user name
297	32	Owner group name
329	8	Device major number
337	8	Device minor number
345	155	Filename prefix

The type flag field can have the following values:

Type flag field

Value	Meaning
'0' or (ASCII NUL)	Normal file
'1'	Hard link
'2'	Symbolic link
'3'	Character special
'4'	Block special
'5'	Directory
'6'	FIFO
'7'	Contiguous file
'g'	global extended header with meta data (POSIX.1-2001)
'x'	extended header with meta data for the next file in the archive (POSIX.1-2001)
'A'–'Z'	Vendor specific extensions (POSIX.1-1988)
All other values	reserved for future standardization

POSIX.1-1988 vendor specific extensions using link flag values 'A'..'Z' partially have a different meaning with different vendors and thus are seen as outdated and replaced by the POSIX.1-2001 extensions that also include a vendor tag.

Type '7' (Contiguous file) is formally marked as reserved in the POSIX standard, but was meant to indicate files which ought to be contiguously allocated on disk. Few operating systems support creating such files explicitly, and hence most TAR programs do not support them, and will treat type 7 files as if they were type 0 (regular). An exception is older versions of GNU tar, when running on the MASSCOMP RTU (Real Time Unix) operating system, which supported an O_CTG flag to the open() function to request a contiguous file; however, that support was removed from GNU tar version 1.24 onwards.

POSIX.1-2001/pax

In 1997, Sun proposed a method for adding extensions to the tar format. This method was later accepted for the POSIX.1-2001 standard. This format is known as extended tar format or pax format. The new tar format allows users to add any type of vendor-tagged vendor-specific enhancements. The following tags are defined by the POSIX standard:

• atime, mtime: all timestamps of a file in arbitrary resolution (most implementations use nanosecond granularity)
• path: path names of unlimited length and character set coding
• linkpath: symlink target names of unlimited length and character set coding
• uname, gname: user and group names of unlimited length and character set coding
• size: files with unlimited size (the historic tar format is 8 GB)
• uid, gid: userid and groupid without size limitation (this historic tar format was is limited to a max. id of 2097151)
• a character set definition for path names and user/group names (UTF-8)

In 2001, the Star program became the first tar to support the new format. In 2004, GNU tar supported the new format, though it does not write them as its default output from the tar program yet.[8]

It's designed so that all implementations able to read the UStar format will be able to read POSIX.1-2001 as well. The only exceptions are files that make use of extended features, such as longer file names. For compatibility, these are encoded in the tar files as special x or g type files. typically under a PaxHeaders.XXXX directory.[9]^:exthdr.name A Pax-supporting implementation would make use of the info, while non supporting ones like 7-Zip would process them as extra files.[10]

Uses

Command syntax

tar [-options] <name of the tar archive> [files or directories which to add into archive]

Basic options:

• -c, --create — create a new archive;
• -a, --auto-compress — additionally compress the archive with a compressor which will be automatically determined by the file name extension of the archive. If the archive's name ends with *.tar.gz then use gzip, if *.tar.xz then use xz, *.tar.zst for Zstandard etc.;
• -r, --append — append files to the end of an archive;
• -x, --extract, --get — extract files from an archive;
• -f, --file — specify the archive's name;
• -t, --list — show a list of files and folders in the archive;
• -v, --verbose — show a list of processed files.

Basic usage

Create an archive file archive.tar from the file README.txt and directory src:

$ tar -cvf archive.tar README.txt src

Extract contents for the archive.tar into the current directory:

$ tar -xvf archive.tar

Create an archive file archive.tar.gz from the file README.txt and directory src and compress it with gzip :

$ tar -cavf archive.tar.gz README.txt src

Extract contents for the archive.tar.gz into the current directory:

$ tar -xvf archive.tar.gz

Tarpipe

A tarpipe is the method of creating an archive on the standard output file of the tar utility and piping it to another tar process on its standard input, working in another directory, where it is unpacked. This process copies an entire source directory tree including all special files, for example:

$ tar cf - srcdir | tar x -C destdir

Software distribution

The tar format continues to be used extensively for open-source software distribution. *NIX-distributions use it in various source- and binary-package distribution mechanisms, with most software source code made available in compressed tar archives.

Limitations

The original tar format was created in the early days of UNIX, and despite current widespread use, many of its design features are considered dated.[11]

Many older tar implementations do not record nor restore extended attributes (xattrs) or Access control lists (ACLs). In 2001, Star introduced support for ACLs and extended attributes, through its own tags for POSIX.1-2001 pax. Bsdtar uses the star extensions to support ACLs.[12] More recent versions of GNU tar support Linux extended attributes, reimplementing star extensions.[13] A number of extensions are reviewed in the filetype manual for BSD tar, tar(5).[12]

Other formats have been created to address the shortcomings of tar.

Operating system support

Unix-like operating systems usually include tools to support tar files, as well as utilities commonly used to compress them, such as gzip and bzip2. BSD-tar has been included in Microsoft Windows since Windows 10 April 2018 Update[14][15], and there are otherwise multiple third party tools available to read and write these formats on Windows.

Tarbomb

"Tarbomb" redirects here. It is not to be confused with zip bomb.

A tarbomb, in hacker slang, is a tar file that contains many files that extract into the working directory. Such a tar file can create problems by overwriting files of the same name in the working directory, or mixing one project's files into another. It is at best an inconvenience to the user, who is obliged to identify and delete a number of files interspersed with the directory's other contents. Such behavior is considered bad etiquette on the part of the archive's creator.

A related problem is the use of absolute paths or parent directory references when creating tar files. Files extracted from such archives will often be created in unusual locations outside the working directory and, like a tarbomb, have the potential to overwrite existing files. However, modern versions of FreeBSD and GNU tar do not create or extract absolute paths and parent-directory references by default, unless it is explicitly allowed with the flag -P or the option --absolute-names. The bsdtar program, which is also available on many operating systems and is the default tar utility on Mac OS X v10.6, also does not follow parent-directory references or symbolic links.[16]

If a user has only a very old tar available, which does not feature those security measures, these problems can be mitigated by first examining a tar file using the command tar tf archive.tar, which lists the contents and allows to exclude problematic files afterwards.

These commands do not extract any files, but display the names of all files in the archive. If any are problematic, the user can create a new empty directory and extract the archive into it—or avoid the tar file entirely. Most graphical tools can display the contents of the archive before extracting them. Vim can open tar archives and display their contents. GNU Emacs is also able to open a tar archive and display its contents in a dired buffer.

Random access

The tar format was designed without a centralized index or table of content for files and their properties for streaming to tape backup devices. The archive must be read sequentially to list or extract files. For large tar archives, this causes a performance penalty, making tar archives unsuitable for situations that often require random access to individual files.

Duplicates

Another issue with tar format is that it allows several (possibly different) files in archive to have identical path and filename. When extracting such archive, usually the latter version of a file overwrites the former.

This can create a non-explicit (unobvious) tarbomb, which technically does not contain files with absolute paths or referring parent directories, but still causes overwriting files outside current directory (for example, archive may contain two files with the same path and filename, first of which is a symlink to some location outside current directory, and second of which is a regular file; then extracting such archive on some tar implementations may cause writing to the location pointed to by the symlink).

Key implementations

Historically, many systems have implemented tar, and many general file archivers have at least partial support for tar (often using one of the implementations below). The history of tar is a story of incompatibilities, known as the "tar wars". Most tar implementations can also read and create cpio and pax (the latter actually is a tar-format with POSIX-2001-extensions).

Key implementations in order of origin:

• Solaris tar, based on the original UNIX V7 tar and comes as the default on the Solaris operating system
• GNU tar is the default on most Linux distributions. It is based on the public domain implementation pdtar which started in 1987. Recent versions can use various formats, including ustar, pax, GNU and v7 formats.
• FreeBSD tar (also BSD tar) has become the default tar on most Berkeley Software Distribution-based operating systems including Mac OS X and Microsoft Windows. The core functionality is available as libarchive for inclusion in other applications. This implementation automatically detects the format of the file and can extract from tar, pax, cpio, zip, jar, ar, xar, rpm and ISO 9660 cdrom images. It also comes with a functionally equivalent cpio command-line interface.
• Schily tar, better known as star, is historically significant as some of its extensions were quite popular. It is the oldest free tar from 1982. It is still maintained.[17]

Additionally, most pax and cpio implementations can read and create many types of tar files.

Suffixes for compressed files

tar archive files usually have the file suffix .tar, e.g., somefile.tar.

A tar archive file contains uncompressed byte streams of the files which it contains. To achieve archive compression, a variety of compression programs are available, such as gzip, bzip2, xz, lzip, lzma, zstd, or compress, which compress the entire tar archive. Typically, the compressed form of the archive receives a filename by appending the format-specific compressor suffix to the archive file name. For example, a tar archive archive.tar, is named archive.tar.gz, when it is compressed by gzip.

Popular tar programs like the BSD and GNU versions of tar support the command line options Z (compress), z (gzip), and j (bzip2) to compress or decompress the archive file upon creation or unpacking. GNU tar from version 1.20 onwards also supports the option --lzma (LZMA). 1.21 added support for lzop with --lzop. 1.22 added support for xz with --xz or -J. 1.23 added support for lzip with --lzip. 1.31 added support for zstd with --zstd.[18] The decompression of these formats is handled automatically if supported filename extensions are used, and compression is handled automatically using the same filename extensions if the option --auto-compress (short form -a) is passed to an applicable version of GNU tar.[8]

MS-DOS's 8.3 filename limitations resulted in additional conventions for naming compressed tar archives. (This practice has declined with FAT offering long filenames.)

Tar archiving is often used together with a compression method, such as gzip, to create a compressed archive. As shown, the combination of the files in the archive are compressed as one unit.

File suffix equivalents[8]

Long	Short
.tar.bz2	.tb2, .tbz, .tbz2, .tz2
.tar.gz	.taz, .tgz
.tar.lz
.tar.lzma	.tlz
.tar.lzo
.tar.xz	.txz
.tar.Z	.tZ, .taZ
.tar.zst	.tzst

See also

• Comparison of file archivers
• Comparison of archive formats
• List of archive formats
• List of Unix commands

References

1. Gilmore, John; Fenlason, Jay (4 February 2019). "Basic Tar Format". gnu.org. and others. Free Software Foundation. Retrieved 17 April 2019.
2. "tar(5) manual page". freebsd.org. FreeBSD. 20 May 2004. Retrieved 2 May 2017.
3. IEEE Std 1003.1-1988, IEEE Standard for Information Technology - Portable Operating System Interface (POSIX)
4. IEEE Std 1003.1-2001, IEEE Standard for Information Technology - Portable Operating System Interface (POSIX)
5. "Blocking'" ftp.gnu.org. Retrieved 28 November 2013.
6. Hoo, James. "Open/Extract TAR File with Freeware on Windows/Mac/Linux". e7z Org. Archived from the original (html) on 6 February 2015. Retrieved 3 September 2019.
7. Kientzle, Tim (1995). Internet File Formats. Coriolis Groups Books. ISBN 978-1-883577-56-8. Retrieved 12 December 2015.
8. "GNU tar 1.32: 8.1 Using Less Space through Compression". GNU. 2019-02-23. Retrieved 2019-08-11.
9. pax – Commands & Utilities Reference, The Single UNIX Specification, Issue 7 from The Open Group
10. "#2116 Tars with pax headers not parsed". 7-Zip / Bugs | SourceForge.
11. Proposed format to replace tar, by the Duplicity utility's developers.
12. tar(5) – FreeBSD File Formats Manual
13. "Extended attributes: the good, the not so good, the bad". Les bons comptes. 15 July 2014. Archived from the original (html) on 14 December 2014. Retrieved 3 September 2019. The extended attributes can be very valuable for storing file metadata (e.g. author="John Smith", subject="country landscape"), in the many cases where you do not want or can’t store this data in the file internal properties.
14. "Announcing Windows 10 Insider Preview Build 17063 for PC". Windows Experience Blog. 2017-12-19. Retrieved 2 July 2018.
15. "Tar and Curl Come to Windows!". 2019-03-22.
16. Man page for "bsdtar", as provided by Apple.
17. star(1) – Linux General Commands Manual
18. Poznyakoff, Sergey (2019-01-02). "tar-1.31 released [stable]". GNU mailing lists. Retrieved 2019-08-06.

External links

• Windows Batch Scripting at Wikibooks