Parallel Compression¶
The classic compressors are single-threaded. On a modern multi-core machine that leaves most of the CPU idle. These tools fix that while producing output in the same format:
| Format | Parallel tool | Decompression parallel? |
|---|---|---|
| gzip | pigz / unpigz |
Partly (format limitation) |
| bzip2 | pbzip2, lbzip2 |
lbzip2: yes, any file; pbzip2: only its own files |
| xz | xz -T0, pixz, pxz |
Yes, for multi-block files |
| zstd | zstd -T0 |
Single-threaded but already very fast |
| lzip | plzip |
Yes |
| 7-Zip | 7z -mmt=on |
Depends on method |
Standalone¶
pigz -p 8 -9 file
unpigz file.gz
lbzip2 -n 8 file
lbzip2 -d file.bz2
xz -T0 -9 file
zstd -T0 -19 file
plzip -n 8 file
With tar¶
tar -I pigz -cf out.tar.gz dir/
tar -I 'pigz -9 -p 8' -cf out.tar.gz dir/
tar -I lbzip2 -cf out.tar.bz2 dir/
tar -I 'xz -T0' -cf out.tar.xz dir/
tar -I 'zstd -T0 -19' -cf out.tar.zst dir/
tar -I pixz -cf out.tpxz dir/
# Decompression too
tar -I unpigz -xf out.tar.gz
tar -I lbzip2 -xf out.tar.bz2
In containers and Kubernetes¶
-T0 and similar "use all cores" options count the CPUs the process can see, which is often the host's core count, not the container's CPU limit. A pod limited to 2 CPUs on a 64-core node will spawn 64 threads and get throttled. Set an explicit count:
zstd -T"$(nproc)" file # nproc respects CPU affinity (cpusets), not CFS quotas
zstd -T2 file # match the pod's CPU limit
In a manifest, pass the limit through as an environment variable with the Downward API (resourceFieldRef: limits.cpu) and use it in the command.
Memory¶
Parallel compression multiplies memory use, roughly per thread. High xz presets with many threads can need gigabytes. Use --memlimit-compress with xz, or lower the thread count.
Ratio trade-off¶
Splitting input into independent chunks costs a little ratio, because matches can't cross chunk boundaries. It's usually under a few percent and well worth the speedup.