Linux Logging - Documentation

Logs are the first place you look when something breaks and the last place you want to discover you had nothing configured. This guide covers how logging actually works on a modern Linux system, how the pieces fit together, and how to configure, ship, rotate, search and troubleshoot logs on both Debian/Ubuntu and RHEL-family (RHEL, Rocky, Alma, Fedora) systems.

Each component has its own page. Start with How Linux Logging Fits Together if you want the big picture first, or use the Quick Reference if you just need a command.

Guide¶

Read in order for the full picture, or jump to the component you need.

How Linux Logging Fits Together — How the kernel ring buffer, systemd-journald, rsyslog, logrotate, auditd and application logs fit together on a modern Linux system, plus distribution defaults.
Where Linux Logs Live — Where log files live on Debian/Ubuntu and RHEL-family systems, from syslog and auth logs to the journal and package manager logs.
Syslog Fundamentals: Facilities, Severities and Formats — Syslog facilities, severity levels, PRI values and the RFC 3164 and RFC 5424 message formats.
The Kernel Ring Buffer and dmesg — Reading the kernel ring buffer with dmesg and journalctl -k, and what to look for: OOM kills, disk errors, segfaults and hardware faults.
systemd-journald and journalctl — Configuring systemd-journald storage and retention, and querying the journal with journalctl: filters, fields, output formats, vacuuming and access control.
rsyslog — rsyslog configuration layout, legacy vs RainerScript syntax, selectors, filters, templates, high-precision timestamps and rate limiting.
Remote and Centralized Logging with rsyslog — Building a central rsyslog server and forwarding clients over UDP, TCP and TLS, with rulesets, per-host files and disk-assisted queues.
Generating Test Messages: logger, systemd-cat and /dev/kmsg — Generating test log messages with logger, systemd-cat and /dev/kmsg to verify routing rules, remote forwarding and alerting.
Firewall Logging (nftables, iptables, firewalld, UFW) — Logging dropped packets with nftables, iptables, firewalld and UFW, and routing firewall messages to their own file with rsyslog.
logrotate — Configuring logrotate: how it runs, writing and testing configs, directive reference, create vs copytruncate, and rotating custom rsyslog outputs.
Login and Audit Records: wtmp, btmp, lastlog and auditd — Reading login history from wtmp, btmp and lastlog, and recording security events with the Linux audit framework (auditd, ausearch, aureport).
Reading and Analyzing Logs (lnav and friends) — Reading and analyzing logs with tail, less, grep, awk and lnav, the Logfile Navigator, including lnav key bindings, filters and SQL queries.
Shipping Logs to a Log Platform — Choosing a log shipper (rsyslog, Vector, Fluent Bit, Grafana Alloy, OpenTelemetry Collector, Filebeat, Logstash) and design tips for centralized logging.
Logging Security, Retention and Hygiene — Log permissions, secrets, time sync, off-host shipping, retention and disk safety for Linux logging.
Troubleshooting Linux Logging — Troubleshooting missing messages, remote logging failures, full disks, logrotate problems, uncompressed rotations and journal issues.
Linux Logging Quick Reference — Cheat sheet of the most-used journalctl, rsyslog, logrotate, dmesg and audit commands.

Guide¶

Further Reading¶