gzip¶

Format: .gz · Algorithm: DEFLATE (LZ77 + Huffman) · First released: 1992 (Jean-loup Gailly and Mark Adler, GNU project)

gzip was written as a patent-free replacement for compress after Unisys began enforcing its LZW patent. It's the universal baseline: every Unix-like system, every language runtime, every browser and HTTP server can read it. It isn't the fastest or the smallest any more, but it's the format you can count on being readable everywhere, indefinitely.

The same DEFLATE algorithm underpins zip, zlib, PNG, and HTTP Content-Encoding: gzip.

Basics¶

gzip file                # -> file.gz, REMOVES file
gzip -k file             # keep the original
gzip -9 file             # best compression (-1 fastest, -6 default)
gzip -d file.gz          # decompress (same as gunzip)
gunzip file.gz           # -> file, removes file.gz
gunzip -k file.gz        # keep the .gz
gzip -c file > out.gz    # write to stdout, leave original untouched
gzip -r dir/             # compress every file in dir/ individually
gzip -l file.gz          # compressed/uncompressed size and ratio
gzip -t file.gz          # test integrity
gzip -v file             # verbose (prints ratio)

gzip -r dir/ produces dir/a.gz, dir/b.gz, … — it does not create a single archive. To compress a directory into one file, use tar: tar czf dir.tar.gz dir/.

Working with .gz files without extracting¶

zcat file.gz             # print contents (same as gzip -dc / gunzip -c)
zless file.gz            # page through
zmore file.gz
zgrep 'ERROR' app.log.gz # grep inside
zgrep -c 'ERROR' *.gz    # count per file
zdiff file1.gz file2.gz  # diff two gzipped files
zcmp file1.gz file2.gz   # byte compare

zcat -f passes uncompressed files through unchanged, which is handy for globbing over a mix of current and rotated logs:

zcat -f /var/log/syslog* | grep sshd

Useful flags¶

Flag	Effect
`-k`, `--keep`	Keep the input file
`-c`, `--stdout`	Write to stdout
`-1` … `-9`, `--fast`, `--best`	Speed vs. ratio (default `-6`)
`-n`, `--no-name`	Don't store original filename/timestamp — gives reproducible output for build artifacts and checksums
`-N`, `--name`	Restore the stored name/timestamp when decompressing
`--rsyncable`	Periodically reset the compressor so small input changes cause small output changes; friendlier to rsync and dedup, slight ratio cost
`-f`, `--force`	Overwrite, compress links, compress to a terminal
`-S .suf`	Use a different suffix

Default compression level from the environment: GZIP=-9 used to work but is deprecated in modern GNU gzip; pass flags explicitly instead.

Concatenation¶

gzip members can be concatenated and still decompress as one stream:

gzip -c part1 > all.gz
gzip -c part2 >> all.gz
zcat all.gz   # outputs part1, then part2

This is handy for appending to compressed logs, but gzip -l reports misleading sizes on multi-member files.

Limitations¶

Single-threaded — use pigz for parallel compression.
32 KB window — DEFLATE can't find matches more than 32 KB back, which caps the ratio on large repetitive data. zstd and xz have far larger windows.
gzip -l stores the uncompressed size modulo 4 GiB, so it's wrong for files over 4 GiB.

pigz¶

Parallel gzip by Mark Adler (2007). Output is standard gzip.

pigz file                # all cores
pigz -p 4 -9 file        # 4 threads, max level
unpigz file.gz
tar -I pigz -cf out.tar.gz dir/

Compression scales across cores; decompression is only partially parallel because of how the DEFLATE format works. See parallel compression.

zopfli¶

Google's (2013) DEFLATE encoder that squeezes out a few extra percent at enormous CPU cost. Output is plain gzip, readable by anything. Useful for static web assets compressed once and served millions of times.

zopfli file              # -> file.gz
zopfli --i50 file        # more iterations, even slower

Other DEFLATE implementations¶

zlib-ng and Intel's igzip (ISA-L) are much faster drop-in implementations of the same format. Many distros now build against zlib-ng.