Is there a way to estimate how much space a file or directory of a given size will take up after being compressed with tar and bzip2?

Question

Due to an imminent distro switch, I would like to backup my home directory. However, my home directory is 29 gigabytes. I would like to know how much space this would take up after being compressed with tar cvjf home.tar /home. Is there a way that I can use to determine the size after compression?

Answer 1

It's not possible to know for certain what size data will compress to without actually compressing it. What you can do get an educated guess based on the content you have in your home directory. I'm not aware of any tools to do this automatically, but it's not a difficult process.

Many modern file formats are already compressed, meaning running it through compression again will give you little to no (or negative) gain. This type of data you're better off skipping the compression and simply copying or archiving it as is. Examples of this would be compressed video (mp4, webm, mov, etc), compressed images (jpeg, png, etc), existing archives (zip, rar, gz, bz2, etc), and more.

Text files will generally compress fairly well, especially if there is a lot of repeated data (ie, log files). You could try sampling a subset of files to see how they compress and use that as a guess or use something like 50% as a rough estimate.

Finally, see what portion of your data is made up of each type and multiple that by your estimated percentage to get an estimate of your final size. For example, if 20GB of your data is compressed data and 9GB is text files, your final compressed data size would probably range from 21GB to 25GB.

Answer 2

Unfortunately, no. The only way to see how large a compressed archive will be, is to create the compressed archive. There is no tool that would do this, as the tool would be doing all the work of the compression program, without writing the final archive, which would be a waste of time.

Perhaps you should consider breaking your data down into manageable chunks and create several archives. This will allow you to break the large amount of time it will take to archive 29 GB, into smaller slices.

Answer 3

The tool is tar (with bzip2 implicitly involved because of j you used) piped to wc (which is a standard (POSIX) tool to count bytes). The following command will print the size in bytes:

tar cj /home | wc -c

The command really does (and I'm citing another answer here) "all the work of the compression program, without writing the final archive, which would be a waste of time"; but if you really want to know then this is the only firm way.

---

You can improve the overall approach like this:

tar cj /home | tee home.tbz2 | wc -c

• If you're lucky and the space you have for home.tbz2 turns out to be enough then you will get no error from tee and the file will end up of size equal to what wc -c will report.
• Otherwise tee will report no space left, yet it will continue writing to its stdout. wc -c will tell you how big the file would be. The actual (incomplete) file will be smaller and you should delete it afterwards.

While using tar with v you may miss a no space left message. Still you can tell what happened by comparing the output you got from wc -c to the actual size of home.tbz2. In Bash you can retrieve the exit status of tee with ${PIPESTATUS[1]}.