Andrew Mercer
on this page

Testing and Benchmarking Compression

Identify an unknown file

file mystery.bin            # reports gzip, bzip2, XZ, Zstandard, Zip, 7-zip, etc.
xxd mystery.bin | head -1   # check the magic bytes yourself
Magic bytes (hex) Format
1f 8b gzip
1f 9d compress (.Z)
1f 1e pack (.z)
42 5a 68 (BZh) bzip2
fd 37 7a 58 5a 00 xz
5d 00 00 legacy LZMA-alone (.lzma, usually)
28 b5 2f fd zstd
04 22 4d 18 lz4 frame
4c 5a 49 50 (LZIP) lzip
89 4c 5a 4f lzop
50 4b 03 04 (PK) zip
37 7a bc af 27 1c 7z
52 61 72 21 (Rar!) RAR
21 3c 61 72 63 68 3e (!<arch>) ar / .deb
30 37 30 37 30 31 (070701) cpio newc
68 73 71 73 (hsqs) SquashFS
75 73 74 61 72 (ustar) at offset 257 tar

Brotli has no magic bytes, so file can't identify raw .br files.

Test integrity

gzip -t file.gz && echo OK
bzip2 -t file.bz2
xz -t file.xz
zstd -t file.zst
lz4 -t file.lz4
lzip -t file.lz
unzip -t file.zip
7z t file.7z
unrar t file.rar
tar tf archive.tar.gz > /dev/null   # also walks the tar structure

Run integrity checks as part of every backup job. A backup that's never been test-read is a hope, not a backup. Better still, periodically restore to a scratch location and compare:

mkdir /tmp/restore && tar xf backup.tar.zst -C /tmp/restore
diff -r /srv/data /tmp/restore/srv/data

Check ratios

gzip -l file.gz
xz -lv file.xz
zstd -lv file.zst
lzip -l file.lz
unzip -v file.zip
7z l file.7z

Benchmark on your own data

Ratios depend heavily on the data, so test on a representative sample.

# zstd's built-in benchmark across levels 1..19
zstd -b1 -e19 sample.tar

# Quick side-by-side
for c in "gzip -6" "bzip2 -9" "xz -6 -T0" "zstd -3 -T0" "zstd -19 -T0" "lz4"; do
  start=$(date +%s.%N)
  size=$($c -c sample.tar | wc -c)
  end=$(date +%s.%N)
  printf '%-14s %12d bytes  %6.2fs\n' "$c" "$size" "$(echo "$end - $start" | bc)"
done

# Decompression speed matters too
hyperfine --warmup 1 \
  'zcat sample.tar.gz > /dev/null' \
  'xzcat sample.tar.xz > /dev/null' \
  'zstdcat sample.tar.zst > /dev/null'

# Many algorithms and levels, in-memory (separate project)
lzbench -ezstd,1,3,19/xz,6/zlib,6/lz4/brotli,11 sample.tar

hyperfine gives repeatable timings with warmups and statistics, for compression and decompression alike.

What compresses well

Data Typical result
Logs, JSON, CSV, SQL dumps, source code Excellent
VM images, database files Good, especially with zstd --long
Executables Moderate
JPEG, PNG, MP3, MP4, WebP Almost nothing
Existing archives (.zip, .docx, .gz) Almost nothing
Encrypted data Nothing

Don't waste CPU on data that won't shrink: zip -n .jpg:.mp4:.png, 7z -mx=0, or plain tar without compression.

See also