Andrew Mercer
on this page

On a current systemd-based distribution there are several cooperating layers:

  ┌────────────┐   printk()      ┌──────────────────┐
  │   Kernel   │ ──────────────▶ │ Kernel ring buf  │──▶ dmesg
  └────────────┘                 │   (/dev/kmsg)    │
                                 └────────┬─────────┘
                                          │
  ┌─────────────────┐  stdout/stderr      ▼
  │ systemd services│ ─────────────▶ ┌──────────────────┐   journalctl
  └─────────────────┘                │ systemd-journald │──▶ /run/log/journal (volatile)
  ┌─────────────────┐  syslog(3)     │                  │   /var/log/journal (persistent)
  │ apps / logger   │ ─────────────▶ │  /dev/log socket │
  └─────────────────┘                └────────┬─────────┘
                                              │ imjournal (RHEL) or
                                              │ syslog socket forward (Debian)
                                              ▼
                                     ┌──────────────────┐
                                     │     rsyslog      │──▶ /var/log/messages, syslog,
                                     │                  │    auth.log, secure, cron …
                                     └────────┬─────────┘
                                              │ omfwd / TCP / TLS
                                              ▼
                                     remote log server / SIEM / Loki / Elasticsearch

  logrotate (systemd timer or cron) rotates, compresses and expires the plain-text files.
  auditd writes its own trail to /var/log/audit/audit.log.
  Applications may also write directly to their own files (nginx, PostgreSQL, BIND …).

Key points:

  • The kernel logs into an in-memory ring buffer (read with dmesg or journalctl -k).
  • systemd-journald collects kernel messages, syslog messages via /dev/log, the stdout/stderr of every systemd unit, and native structured journal entries. It stores them in an indexed binary format.
  • rsyslog (optional on some distros now) receives messages from journald and writes the classic plain-text files under /var/log, and is the usual tool for forwarding to remote servers.
  • logrotate keeps plain-text files from filling the disk. The journal manages its own size.
  • Application logs that bypass syslog entirely (web servers, databases, Java apps) still need rotation and, ideally, shipping.

Distribution defaults worth knowing

Distribution rsyslog installed by default Persistent journal by default
Debian 12+ No Yes (/var/log/journal exists)
Ubuntu (server) Yes Yes
RHEL / Rocky / Alma 8–10 Yes Often volatile only (no /var/log/journal dir)
Fedora No (recent releases) Yes
Arch No Yes

Defaults move between releases, so verify on the host in front of you:

systemctl is-active rsyslog
ls -ld /var/log/journal /run/log/journal 2>/dev/null
journalctl --header | grep -i 'file path'

On a Debian 12+ box without rsyslog, there is no /var/log/syslog or /var/log/auth.log. Everything is in the journal unless you install rsyslog (apt install rsyslog).