Compression in Rust¶

For CLI tools and services that need to read or write these formats, the Rust ecosystem has mature crates for almost everything.

Crates¶

Crate	Formats	Notes
`flate2`	gzip, zlib, raw DEFLATE	Pure-Rust `miniz_oxide` backend by default; `zlib-rs` or `zlib-ng` backends for speed
`zstd`	zstd	Bindings to libzstd; multithreaded encoder (`zstdmt` feature), dictionaries
`xz2` / `liblzma`	xz, lzma	Bindings to liblzma; `liblzma` is the maintained fork
`lzma-rs`	xz, lzma (decode)	Pure Rust
`bzip2`	bzip2	Bindings, with a pure-Rust backend option in newer versions
`lz4_flex`	lz4	Pure Rust, fast
`brotli`	brotli	Pure Rust
`tar`	tar	Read/write over any `Read`/`Write`
`zip`	zip	Read/write, including AES and zstd/bzip2/deflate64 methods (feature flags)
`sevenz-rust2`	7z	Read/write, including AES
`async-compression`	Most of the above	`tokio`/`futures` async wrappers

The common pattern: compressors implement Read (decoders) and Write (encoders), so they wrap files, sockets, or each other, just like a shell pipeline.

Streaming .tar.zst¶

use std::fs::File;
use std::io;

fn create(src: &str, out: &str) -> io::Result<()> {
    let file = File::create(out)?;
    let mut enc = zstd::Encoder::new(file, 19)?;
    enc.multithread(std::thread::available_parallelism()?.get() as u32)?;
    let mut tar = tar::Builder::new(enc);
    tar.append_dir_all(".", src)?;
    tar.into_inner()?.finish()?;
    Ok(())
}

fn extract(archive: &str, dest: &str) -> io::Result<()> {
    let dec = zstd::Decoder::new(File::open(archive)?)?;
    tar::Archive::new(dec).unpack(dest)?;
    Ok(())
}

multithread requires the zstdmt feature on the zstd crate. Call finish() on encoders — dropping them without it can leave a truncated stream.

gzip¶

use flate2::{read::GzDecoder, write::GzEncoder, Compression};
use std::io::{copy, BufReader};
use std::fs::File;

fn gzip(input: &str, output: &str) -> std::io::Result<()> {
    let mut reader = BufReader::new(File::open(input)?);
    let mut enc = GzEncoder::new(File::create(output)?, Compression::best());
    copy(&mut reader, &mut enc)?;
    enc.finish()?;
    Ok(())
}

fn gunzip(input: &str, output: &str) -> std::io::Result<()> {
    let mut dec = GzDecoder::new(File::open(input)?);
    copy(&mut dec, &mut File::create(output)?)?;
    Ok(())
}

Use MultiGzDecoder for files made of concatenated gzip members.

Pick the codec from the extension¶

use std::{fs::File, io::{self, Read}, path::Path};

fn open_decompressed(path: &Path) -> io::Result<Box<dyn Read>> {
    let f = File::open(path)?;
    Ok(match path.extension().and_then(|e| e.to_str()) {
        Some("gz")  => Box::new(flate2::read::MultiGzDecoder::new(f)),
        Some("zst") => Box::new(zstd::Decoder::new(f)?),
        Some("xz")  => Box::new(xz2::read::XzDecoder::new(f)),
        Some("bz2") => Box::new(bzip2::read::BzDecoder::new(f)),
        _           => Box::new(f),
    })
}

Sniffing magic bytes is more robust than trusting extensions.

Safety when extracting¶

tar::Archive::unpack refuses entries that escape the destination, but if you iterate entries and write them yourself, validate each path: reject absolute paths and .. components.
With zip, use ZipFile::enclosed_name() rather than the raw name.
Cap total decompressed size and entry count when handling untrusted input, to defend against decompression bombs.

See gotchas.