For more than a handful of hosts, logs belong in a central, searchable system: Elasticsearch/OpenSearch, Grafana Loki, Graylog, Splunk, or a cloud service. rsyslog can forward there, but a dedicated shipper usually handles parsing, buffering and multiple outputs more cleanly.
| Shipper | Notes |
|---|---|
rsyslog (omfwd, omelasticsearch, omkafka, omhttp) |
Already installed; good for syslog-native pipelines |
| Vector | Fast single binary; reads journald, files, syslog; rich transforms (VRL); many sinks |
| Fluent Bit | Lightweight, Kubernetes-native; journald and tail inputs |
| Grafana Alloy | Grafana's collector (successor to Promtail) for Loki, plus metrics and traces |
| OpenTelemetry Collector | journald, filelog and syslog receivers; vendor-neutral OTLP output |
| Filebeat / Elastic Agent | Native fit for the Elastic stack |
| Logstash | Heavy but powerful parsing/enrichment tier in front of Elasticsearch |
Design tips:
- Read from journald where possible. It gives you structured fields (unit, PID, priority, boot ID) for free rather than reparsing text.
- Buffer on disk at the edge so an outage at the collector doesn't lose data.
- Normalize timestamps to UTC with sub-second precision, and keep the hostname, unit/program and severity as indexed fields.
- Use labels/indices sparingly. High-cardinality labels (request IDs, user IDs) hurt Loki in particular; keep them in the log body.
- Drop noise at the edge (health checks, debug chatter) rather than paying to store it.
- Prefer a message bus (Kafka, RabbitMQ, Redis streams) between shippers and the indexer when ingest is spiky.