Compression Pipelines¶

Every tool in this section reads stdin and writes stdout, so they chain with pipes. That avoids temporary files, which matters when the data is larger than the free disk space.

Copy a directory tree over SSH¶

# Push
tar cf - src_dir/ | zstd -T0 | ssh user@host 'zstd -d | tar xf - -C /dest'

# Pull
ssh user@host 'tar cf - -C /var/www site | zstd -T0' | zstd -d | tar xf -

# Simpler: SSH's own zlib compression, fine on slow links
tar cf - src_dir/ | ssh -C user@host 'tar xf - -C /dest'

# Save to a remote file instead of extracting
tar cf - src_dir/ | zstd -T0 | ssh user@host 'cat > /backups/src.tar.zst'

Pick the compressor for the link. On a fast LAN, heavy compression becomes the bottleneck and makes the transfer slower. lz4, zstd -1, or no compression wins there. On slow WAN links, higher zstd levels or --adapt pay off.

Progress bars¶

tar cf - big_dir/ | pv -s "$(du -sb big_dir | cut -f1)" | zstd -T0 > big.tar.zst
pv big.tar.zst | zstd -d | tar xf -

zstd, xz -v and pigz also print their own progress when writing to a file.

Split into chunks and rejoin¶

Useful for size-limited storage or upload targets:

tar cf - big_dir/ | zstd -T0 | split -b 1G -d - big.tar.zst.part-
cat big.tar.zst.part-* | zstd -d | tar xf -

# Verify the parts before deleting the source
cat big.tar.zst.part-* | zstd -t

Database dumps¶

# PostgreSQL
pg_dump mydb | zstd -T0 > mydb.sql.zst
zstd -dc mydb.sql.zst | psql mydb
pg_dump -Fc -Z zstd mydb > mydb.dump          # PostgreSQL 16+: built-in zstd

# MariaDB / MySQL
mysqldump --single-transaction mydb | zstd -T0 > mydb.sql.zst
zstd -dc mydb.sql.zst | mysql mydb
mariadb-backup --backup --stream=xbstream | zstd -T0 > full.xb.zst

Logs and command output¶

journalctl --since today | gzip > journal-$(date +%F).log.gz
kubectl logs deploy/myapp --since=24h | zstd > myapp-$(date +%F).log.zst
some-long-job 2>&1 | zstd > job.log.zst

Kubernetes pods¶

kubectl cp runs tar inside the container and fails if the image doesn't have it. For large copies, stream it yourself:

# Out of a pod
kubectl exec -n ns pod -- tar cf - /data | zstd -T0 > pod-data.tar.zst

# Into a pod
zstd -dc pod-data.tar.zst | kubectl exec -i -n ns pod -- tar xf - -C /

If the image has no tar, use kubectl debug with an image that does, sharing the target container's process namespace.

Disk images¶

# Compress a block device image, skipping zeroes efficiently with zstd
sudo dd if=/dev/sdX bs=4M status=progress | zstd -T0 > disk.img.zst
zstd -dc disk.img.zst | sudo dd of=/dev/sdX bs=4M status=progress

Encrypted backups¶

Compress, then encrypt, in one pipeline:

tar cf - /srv/data | zstd -T0 | age -r age1... > data.tar.zst.age

See encryption.