zstd¶

Format: .zst · Algorithm: LZ77 + Finite State Entropy (tANS) + Huffman · First released: 2015, 1.0 in 2016 (Yann Collet, Facebook/Meta) · Spec: RFC 8878

Zstandard is the modern default. One format covers nearly the whole speed/ratio spectrum: low levels are close to lz4 speed, high levels approach xz ratios, and decompression is very fast at every level. It has been adopted for distro packages, the Linux kernel and initramfs, Btrfs/ZFS, systemd-journald, container image layers, and HTTP.

Basics¶

zstd file                # -> file.zst, KEEPS file
zstd --rm file           # remove source after success
zstd -d file.zst         # decompress (same as unzstd)
unzstd file.zst
zstd -c file > out.zst   # to stdout
zstd -o out.zst file     # explicit output name
zstd -3 file             # default level
zstd -19 file            # high compression
zstd --ultra -22 file    # maximum (needs --ultra above 19; lots of memory)
zstd --fast=5 file       # negative levels: faster than -1, lower ratio
zstd -T0 file            # use all cores
zstd -l file.zst         # list info
zstd -lv file.zst        # verbose info
zstd -t file.zst         # test integrity
zstd -r dir/             # compress each file in dir/ individually (not an archive)

zstd keeps the original file by default — the opposite of gzip, bzip2 and xz. Add --rm if you expect the source to disappear, especially in disk-cleanup scripts.

Levels¶

Working with .zst files without extracting¶

zstdcat file.zst
zstdless file.zst
zstdgrep 'pattern' file.zst

With tar¶

tar --zstd -cvf archive.tar.zst src_dir/          # GNU tar 1.31+
tar caf archive.tar.zst src_dir/                  # pick compressor from extension
tar -I 'zstd -T0 -19' -cf archive.tar.zst src_dir/
tar xf archive.tar.zst                            # auto-detected on extract
tar --zstd -tvf archive.tar.zst                   # list

Long-distance matching¶

--long enlarges the match window (128 MiB by default with --long, or --long=N for a 2^N-byte window). It gives big wins on large inputs where repetition is far apart: VM images, database dumps, tarballs of many similar files.

zstd -T0 -19 --long=31 big.tar

Decompressing a file made with a window larger than 128 MiB requires telling the decoder it may use that much memory:

zstd -d --long=31 big.tar.zst
# or
zstd -d --memory=2048MB big.tar.zst

Adaptive mode¶

--adapt adjusts the level on the fly based on how fast output is consumed — useful when piping over a link of unknown speed:

tar cf - /data | zstd -T0 --adapt | ssh host 'cat > data.tar.zst'

Dictionaries¶

For many small, similar files (JSON records, log lines, small blobs), a trained dictionary dramatically improves the ratio, because each tiny file otherwise has no history to match against:

zstd --train samples/*.json -o json.dict
zstd -D json.dict record.json
zstd -D json.dict -d record.json.zst

The same dictionary is needed to decompress, so version it alongside the data.

Binary patches¶

zstd can produce compact diffs between two versions of a large file:

zstd --patch-from=v1.bin v2.bin -o v2.patch.zst
zstd -d --patch-from=v1.bin v2.patch.zst -o v2.bin

Other useful flags¶

Where you'll see it¶

• Distro packages (Arch, Fedora RPM payloads, Debian/Ubuntu .deb payloads on newer releases)
• Kernel and initramfs compression
• Btrfs compress=zstd, ZFS compression=zstd, SquashFS -comp zstd
• systemd-journald
• OCI container image layers
• HTTP Content-Encoding: zstd

See compression in the stack.

See also¶

• lz4 — even faster, lower ratio
• xz — the format zstd most often replaces
• Parallel compression and pipelines