Logstash: Comprehensive Reference Guide¶

1. Architecture¶

Logstash pipelines have three stages, all defined in a config file:

input { }   # where events come from
filter { }  # transform/enrich events
output { }  # where events go

Events flow through in-memory queues between stages by default. For durability across restarts/crashes, enable the persistent queue (queue.type: persisted in logstash.yml) — without it, in-flight events are lost on a crash.

Key daemons/files:

Because files in conf.d/ are concatenated, naming convention matters — hence the 10-, 11- prefixes in your notes (input configs low, filters middle, output high, so ordering is predictable regardless of alphabetical surprises).

2. Multiple Pipelines¶

Instead of one giant conf.d/, pipelines.yml lets you isolate workloads (e.g., separate DLQ-heavy pipeline from high-volume nginx access logs) so a slow filter in one doesn't back-pressure the other:

- pipeline.id: nginx
  path.config: "/etc/logstash/conf.d/nginx/*.conf"
  pipeline.workers: 4

- pipeline.id: dlq-reprocess
  path.config: "/etc/logstash/conf.d/dlq/*.conf"
  pipeline.workers: 2

3. Inputs¶

Common ones you'll actually use:

input {
  beats {
    port => 5044
  }
  http {
    port => 8080
  }
  kafka {
    bootstrap_servers => "kafka:9092"
    topics => ["app-logs"]
  }
}

Filebeat note (your nginx section is outdated): prospectors: was deprecated in Filebeat 6.x and removed in 7.x. Modern config uses filebeat.inputs::

filebeat.inputs:
  - type: filestream
    id: nginx-access
    paths:
      - /var/log/nginx/access.log
    fields:
      log_type: nginx-access
    fields_under_root: true

document_type is also gone (removed in 6.0+, all docs go to _doc). Use fields + a Logstash conditional on that field instead, as above.

4. Grok Patterns¶

Custom pattern files (your approach is correct and still standard):

mkdir -p /etc/logstash/patterns
chown logstash: /etc/logstash/patterns/nginx

# /etc/logstash/patterns/nginx
NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent}

filter {
  if [log_type] == "nginx-access" {
    grok {
      patterns_dir => ["/etc/logstash/patterns"]
      match => { "message" => "%{NGINXACCESS}" }
    }
    date {
      match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
      target => "@timestamp"
      remove_field => ["timestamp"]
    }
    useragent {
      source => "agent"
      target => "user_agent"
    }
    geoip {
      source => "clientip"
    }
    mutate {
      convert => { "response" => "integer", "bytes" => "integer" }
    }
  }
}

Notes on additions vs. your original:
- date filter — without it, @timestamp is when Logstash processed the event, not when nginx logged it. For anything ILM-driven or used in incident timelines, this matters a lot.
- useragent/geoip — cheap enrichment, commonly wanted on access logs; drop if you don't need it (geoip in particular adds CPU cost per event).
- mutate { convert } — grok fields are strings by default; response/bytes should be numeric for aggregations/visualizations in Kibana.
- Test grok patterns against sample lines with the Grok Debugger or bin/logstash -e before deploying — a bad pattern silently tags events _grokparsefailure instead of erroring loudly.

5. Ruby Filter — Your DLQ Snippet, Cleaned Up¶

Your working version:

if "dlq" in [tags] {
  ruby {
    id => "prefix_dlq_message"
    code => '
      event.to_hash.each { |k,v|
        event.set("[error][#{k}]",v)
        event.remove(k)
      }'
  }
}

The real risk here (as you'd already noted from your ChatGPT sessions, correctly) is mutating event.to_hash while iterating it — to_hash in the JRuby event API returns a snapshot, so in practice this doesn't infinite-loop, but it's fragile and will silently misbehave if the internal implementation changes. Snapshot the keys first and exclude metadata fields so you don't nest @timestamp/tags/@version under [error] (which breaks index templates expecting @timestamp at the root):

if "dlq" in [tags] {
  ruby {
    id => "prefix_dlq_message"
    code => '
      preserve = ["@timestamp", "@version", "tags", "@metadata"]
      event.to_hash.keys.each { |k|
        next if preserve.include?(k)
        event.set("[error][#{k}]", event.get(k))
        event.remove(k)
      }'
  }
}

Better alternative — skip Ruby entirely. If the goal is just "namespace the whole payload under error," a prune + mutate { rename } chain is more idiomatic Logstash and doesn't run arbitrary code per-event:

if "dlq" in [tags] {
  mutate {
    rename => { "message" => "[error][message]" }
  }
}

— only reach for ruby {} when you genuinely need dynamic/programmatic field manipulation that the built-in filters can't express. Ruby filters are also the easiest way to accidentally tank pipeline throughput, since the code runs per event with no batching.

Also worth knowing: Logstash has a built-in Dead Letter Queue feature (dead_letter_queue.enable: true in logstash.yml), separate from your manual "dlq" in [tags] convention. The built-in DLQ captures events that fail at the output stage (e.g., Elasticsearch mapping conflicts) and lets you reprocess them via a dead_letter_queue input later. Your tag-based approach looks like it's for filter-stage error handling instead, which the built-in DLQ doesn't cover — so keeping both patterns is reasonable, just worth being explicit in comments about which failure mode each one is for.

6. Common Filters Cheat Sheet¶

filter {
  mutate {
    rename => { "old_field" => "new_field" }
    remove_field => ["unwanted"]
    lowercase => ["some_field"]
  }

  date {
    match => ["timestamp", "ISO8601"]
  }

  kv {
    source => "message"
    field_split => " "
  }

  fingerprint {
    source => ["clientip", "request"]
    target => "[@metadata][fingerprint]"
    method => "SHA1"
  }
}

7. Conditionals¶

if [type] == "nginx-access" and [response] >= 500 {
  mutate { add_tag => ["nginx_5xx"] }
} else if "dlq" in [tags] {
  # ...
}

[@metadata] fields are useful for values you need mid-pipeline (like a dynamic index name) but don't want indexed into Elasticsearch — they're dropped automatically before output.

8. Output¶

output {
  if "nginx_5xx" in [tags] {
    elasticsearch {
      hosts => ["https://es-node:9200"]
      index => "nginx-errors-%{+YYYY.MM.dd}"
      ilm_enabled => true
      ilm_rollover_alias => "nginx-errors"
      ilm_policy => "nginx-errors-policy"
    }
  } else {
    elasticsearch {
      hosts => ["https://es-node:9200"]
      index => "logs-generic-%{+YYYY.MM.dd}"
    }
  }
}

Given the ILM naming work you've been doing, worth standardizing the index => naming in these output blocks to match your retention-duration convention up front rather than backfilling aliases later.

9. Performance Tuning¶

• pipeline.workers — default is CPU core count; raise for CPU-bound filters (grok, geoip), not for I/O-bound ones.
• pipeline.batch.size / pipeline.batch.delay — larger batches improve ES bulk throughput at the cost of latency.
• Order filters cheapest-to-most-expensive, and put conditionals around expensive filters (geoip, ruby, dissect on large strings) so they only run on the events that need them.
• grok is regex-based and slow relative to dissect — if a log format is fixed-delimiter (not free text), use dissect first and grok only for the free-text remainder.
• Persistent queue trades throughput for durability — size queue.max_bytes based on how much backlog you're willing to buffer during an ES outage.

10. Monitoring & Troubleshooting¶

• GET _cat/plugins and the Logstash monitoring API (http://localhost:9600/_node/stats) for pipeline health, events-per-second, and queue depth.
• _grokparsefailure / _dateparsefailure tags — grep for these in Kibana to find silently-dropped enrichment.
• bin/logstash -f pipeline.conf --config.test_and_exit — validate config syntax before reload.
• bin/logstash -f pipeline.conf --config.reload.automatic — hot-reload during development.

11. Reference¶

• Logstash filter plugin index
• Grok Debugger
• Your original nginx/DLQ notes were pulled from a DigitalOcean tutorial and ChatGPT sessions — both technically sound as a starting point; the changes above mainly close gaps around timestamp accuracy, field typing, and the deprecated Filebeat prospectors syntax.