Compression in the Stack¶

Compression isn't only something you run by hand. It's configured throughout a typical Linux platform, and knowing where lets you tune it.

logrotate¶

/var/log/myapp/*.log {
    daily
    rotate 14
    compress
    delaycompress            # leave the newest rotated file uncompressed
    compresscmd /usr/bin/zstd
    compressext .zst
    compressoptions -T0 -10
    uncompresscmd /usr/bin/unzstd
    missingok
    notifempty
}

delaycompress helps when the application may still write to the just-rotated file. The default compressor is gzip.

systemd-journald¶

Journal objects above a size threshold are compressed automatically (zstd on modern builds, previously xz or lz4). Controlled by Compress= in /etc/systemd/journald.conf (yes, no, or a byte threshold).

journalctl --disk-usage

HTTP¶

Servers compress responses on the fly with Content-Encoding: gzip, br (brotli), or zstd.

gzip on;
gzip_comp_level 5;
gzip_types text/plain text/css application/json application/javascript image/svg+xml;
gzip_min_length 1024;
gzip_static on;          # serve pre-compressed .gz files if present
gzip_vary on;

Brotli and zstd need extra nginx modules. Pre-compressing static assets at build time (brotli -q 11, zopfli, zstd -19) gives the best ratio without per-request CPU. Avoid compressing secrets mixed with attacker-controlled content in the same response (the BREACH class of attacks).

Container images¶

OCI image layers are compressed tarballs. zstd layers pull and unpack noticeably faster than gzip:

podman push --compression-format zstd --compression-level 10 image registry/image:tag
docker buildx build --output type=image,name=registry/image:tag,compression=zstd,push=true .

Older container runtimes can't read zstd layers, so check your cluster's containerd or CRI-O version first. zstd:chunked (in podman/CRI-O) additionally allows partial pulls of changed files.

Databases¶

• PostgreSQL: pg_dump -Fc writes a compressed custom-format dump; PostgreSQL 16+ accepts -Z zstd and -Z lz4 as well as gzip. Large column values are TOAST-compressed with pglz or lz4 (default_toast_compression). WAL can be compressed with wal_compression.
• MariaDB/MySQL: InnoDB page compression (PAGE_COMPRESSED=1) and table compression; stream backups with mariadb-backup --stream=xbstream | zstd.
• See pipelines for dump commands.

Elasticsearch and OpenSearch¶

• index.codec: best_compression trades some CPU for noticeably smaller stored fields. It applies to newly written segments; force-merge to apply it to existing data.
• Snapshot repositories compress metadata by default; compress: true on the repository extends this.
• Transport and HTTP compression are separate settings.

Kernel and boot¶

The kernel image and initramfs can be compressed with gzip, xz, lz4, or zstd:

Rebuild the initramfs afterwards. See cpio for inspecting images.

Memory: zram and zswap¶

• zram creates a compressed block device in RAM, usually used as swap. It's the default on Fedora and common on low-memory nodes.
• zswap is a compressed cache in front of a regular swap device.

zramctl                               # devices, algorithm, ratio
cat /sys/block/zram0/comp_algorithm   # [lzo-rle] lz4 zstd ...
cat /sys/module/zswap/parameters/compressor

Filesystems¶

Btrfs, ZFS and others can compress transparently. See SquashFS and compressed filesystems.

Packages¶

See ar.

See also¶

• zstd
• Overview