Remote and Centralized Logging with rsyslog

Terminology used here:

Client — the host that sends its logs.
Server (log host / collector) — the host that receives and stores logs.

Transport choices¶

Transport	Port	Pros	Cons
UDP	514	Universal (routers, switches, appliances), no connection overhead	Lossy, unordered, unauthenticated, plaintext
TCP	514 (or 601)	Reliable delivery while connected, works with queues	Plaintext, unauthenticated
TCP + TLS	6514	Encrypted, mutually authenticated	Requires certificates and `rsyslog-gnutls` or `rsyslog-openssl`
RELP	2514 (convention)	Application-level acknowledgements, no loss on connection drop	Both ends need `rsyslog-relp`

Use TLS for anything crossing an untrusted network. Use UDP only for devices that can't do anything else, and keep it on a management network.

Server: receive and separate remote logs (recommended layout)¶

The cleanest pattern is to bind network inputs to their own ruleset. Remote messages are then processed only by that ruleset and never fall through into the server's local /var/log/messages or /var/log/syslog.

/etc/rsyslog.d/10-remote-server.conf:

# Load network listeners
module(load="imtcp")
module(load="imudp")

# One directory per sending host, one file per program
template(name="RemotePerHost" type="string"
         string="/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log")

ruleset(name="remote") {
    # Optional: route specific devices to a dedicated file
    if $fromhost-ip == "192.0.2.10" then {
        action(type="omfile" file="/var/log/remote/router1.log")
        stop
    }

    action(type="omfile" dynaFile="RemotePerHost"
           dirCreateMode="0750" fileCreateMode="0640"
           dirOwner="root" dirGroup="adm" fileOwner="root" fileGroup="adm")
    stop
}

input(type="imtcp" port="514" ruleset="remote")
input(type="imudp" port="514" ruleset="remote")

Validate, restart and confirm it's listening:

sudo rsyslogd -N1
sudo systemctl restart rsyslog
sudo ss -ltnup | grep ':514'
# LISTEN 0 25   0.0.0.0:514  0.0.0.0:*  users:(("rsyslogd",pid=3341,fd=6))
# UNCONN 0 0    0.0.0.0:514  0.0.0.0:*  users:(("rsyslogd",pid=3341,fd=5))

Open the firewall, ideally only to your client subnets:

# firewalld (RHEL family)
sudo firewall-cmd --permanent --zone=public \
  --add-rich-rule='rule family="ipv4" source address="192.0.2.0/24" port port="514" protocol="tcp" accept'
sudo firewall-cmd --permanent --zone=public \
  --add-rich-rule='rule family="ipv4" source address="192.0.2.0/24" port port="514" protocol="udp" accept'
sudo firewall-cmd --reload

# Or open broadly (lab only):
# sudo firewall-cmd --permanent --add-service=syslog   # 514/udp
# sudo firewall-cmd --permanent --add-port=514/tcp

# UFW (Debian/Ubuntu)
sudo ufw allow from 192.0.2.0/24 to any port 514 proto tcp
sudo ufw allow from 192.0.2.0/24 to any port 514 proto udp

SELinux: 514/tcp, 514/udp and 6514/tcp are normally already labelled syslogd_port_t. If you use a non-standard port, label it:

sudo semanage port -l | grep syslog
sudo semanage port -a -t syslogd_port_t -p tcp 10514

Why remote logs used to "leak" into local files. A classic legacy setup puts :fromhost-ip, isequal, "x.x.x.x" /var/log/router.log followed by & ~ into /etc/rsyslog.conf. If that rule appears after the default rules (for example, after $IncludeConfig /etc/rsyslog.d/*.conf, which pulls in 50-default.conf), the message has already been written to syslog, messages, kern.log, user.log etc. before the discard happens. It can look like one device behaves and another doesn't, simply because their messages use different facilities (a Linux router sending kern messages matches more default rules). The fix is either to put the filter in an early drop-in (/etc/rsyslog.d/10-…conf) followed by stop, or, better, to bind the network input to a dedicated ruleset as shown above.

Client: forward reliably with a disk-assisted queue¶

A bare *.* @@logserver:514 works, but if the server is down, messages are buffered only briefly in memory and then dropped. Use an action queue so messages spool to disk and are delivered when the server returns.

/etc/rsyslog.d/90-forward.conf:

# Where queue spool files live (often already set in rsyslog.conf)
global(workDirectory="/var/lib/rsyslog")

*.* action(
    type="omfwd"
    target="logs.example.com"
    port="514"
    protocol="tcp"
    template="RSYSLOG_SyslogProtocol23Format"   # RFC 5424 on the wire
    queue.type="LinkedList"
    queue.filename="fwd_logs_example"           # setting a filename makes it disk-assisted
    queue.maxDiskSpace="1g"
    queue.saveOnShutdown="on"
    action.resumeRetryCount="-1"                # retry forever
    action.resumeInterval="30"
)

sudo rsyslogd -N1 && sudo systemctl restart rsyslog

The legacy one-liner for comparison (@ = UDP, @@ = TCP):

*.* @@logs.example.com:514

You can forward a subset: auth,authpriv.* action(type="omfwd" …) or *.warning action(…).

Encrypting with TLS¶

Install the TLS driver on both ends (rsyslog-gnutls on Debian/RHEL; rsyslog-openssl is the alternative). You need a CA certificate on both ends plus a certificate/key pair for each host. A private CA is fine.

Server, /etc/rsyslog.d/10-remote-tls.conf:

global(
    DefaultNetstreamDriver="gtls"
    DefaultNetstreamDriverCAFile="/etc/rsyslog.d/tls/ca.pem"
    DefaultNetstreamDriverCertFile="/etc/rsyslog.d/tls/server-cert.pem"
    DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/tls/server-key.pem"
)

module(load="imtcp"
       StreamDriver.Name="gtls"
       StreamDriver.Mode="1"                  # TLS only
       StreamDriver.AuthMode="x509/name"      # verify client certificate names
       PermittedPeer=["*.example.com"])

input(type="imtcp" port="6514" ruleset="remote")

Client:

global(
    DefaultNetstreamDriver="gtls"
    DefaultNetstreamDriverCAFile="/etc/rsyslog.d/tls/ca.pem"
    DefaultNetstreamDriverCertFile="/etc/rsyslog.d/tls/client-cert.pem"
    DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/tls/client-key.pem"
)

*.* action(
    type="omfwd" target="logs.example.com" port="6514" protocol="tcp"
    StreamDriver="gtls" StreamDriverMode="1"
    StreamDriverAuthMode="x509/name"
    StreamDriverPermittedPeers="logs.example.com"
    queue.type="LinkedList" queue.filename="fwd_tls" queue.maxDiskSpace="1g"
    queue.saveOnShutdown="on" action.resumeRetryCount="-1"
)

Key files must be readable by the rsyslog process (root on most distros; the syslog user on Ubuntu). Debug handshake failures with journalctl -u rsyslog and openssl s_client -connect logs.example.com:6514 -CAfile ca.pem.

Network devices¶

Routers, switches, firewalls and appliances are usually configured from their own CLI or web UI: point them at the log server on 514/udp (or TCP/TLS if supported) and assign a distinct facility such as local4. On the server, route by $fromhost-ip or facility as shown above.

Verifying end to end¶

On the client:

logger -p user.info "remote logging test from $(hostname)"
logger -p authpriv.crit "remote logging test: authpriv.crit"

On the server:

sudo tail -F /var/log/remote/*/*.log
sudo tcpdump -ni any port 514 -A     # see raw syslog on the wire (non-TLS only)