Compression and Archiving¶

This section covers the tools used to shrink and bundle files on Linux, from the ones you'll reach for today to the formats you'll still find in old backups and FTP mirrors. Each tool has its own page; this page explains the concepts, compares the options, and links to everything.

Core concepts¶

Archiving vs. compression¶

Unix traditionally splits two jobs between two kinds of tool:

The single-stream compressors compress one file. To compress a directory, archive it with tar first and compress the tar stream. That's what .tar.gz, .tar.zst and friends are.

Solid vs. per-file compression¶

• Solid (tarballs, solid 7z): the whole archive is compressed as one stream. Better ratio, because redundancy across files is exploited. Downside: extracting the last file means decompressing everything before it, and corruption can damage everything after the bad byte.
• Per-file (zip): each member is compressed independently. Fast random access and better corruption isolation, at the cost of ratio on many small similar files.

The trade-off triangle¶

Every compressor trades ratio, compression speed, and decompression speed/memory. Level flags (-1 … -9) move you along one algorithm's curve; choosing a different algorithm moves the whole curve. Today zstd dominates most of that curve, which is why it has become the default in so many places.

Choosing a tool¶

Relative characteristics (the shape of the curve, not exact numbers — benchmark on your own data):

The headline: zstd decompresses fast at every level, while xz and bzip2 are slow to decompress no matter how they were compressed.

Modern tools¶

Single-stream compressors¶

Archivers and archive formats¶

Historical tools¶

Formats you'll mostly meet when digging through old archives, BBS-era software collections, or legacy systems.

A short history¶

• 1970s: Unix gets ar for libraries, then tar (Version 7, 1979) for tapes and cpio for PWB/UNIX. Compression is pack, simple Huffman coding.
• 1984–85: Terry Welch publishes LZW; compress adopts it and becomes the Unix standard .Z format.
• Late 1980s: The DOS world fragments into ARC, ZIP (Phil Katz, 1989), LHA, ZOO, then ARJ and RAR in the early '90s.
• 1992: Unisys starts enforcing its LZW patent. The GNU project releases gzip, built on the patent-free DEFLATE algorithm, and .tar.gz becomes the lingua franca of free software.
• 1996: bzip2 trades speed for a better ratio and gradually takes over source tarballs.
• 1999–2009: Igor Pavlov's 7-Zip introduces LZMA; LZMA Utils and then xz (2009) bring it to Unix. Kernel and distro tarballs move to .tar.xz.
• 2011–2016: Speed matters as much as ratio: lz4 (2011), brotli for the web, then zstd (2016), which covers nearly the whole speed/ratio range.
• 2018 onward: zstd becomes the default across distro packages, kernels, initramfs, filesystems, journald, and container image layers.
• 2024: The xz-utils backdoor (CVE-2024-3094) is a high-profile reminder that compression libraries sit deep in the supply chain — see xz.

File extensions at a glance¶

Cross-cutting topics¶