1. Architecture¶

input { }   # where events come from
filter { }  # transform/enrich events
output { }  # where events go

Events pass through an in-memory queue by default. Enable queue.type: persisted in logstash.yml for durability across restarts/crashes — otherwise in-flight events are lost.

2. Filters Reference¶

filter {
  grok {
    patterns_dir => ["/etc/logstash/patterns"]
    match => { "message" => "%{NGINXACCESS}" }
  }
  date {
    match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
    target => "@timestamp"
  }
  mutate {
    rename => { "old_field" => "new_field" }
    convert => { "response" => "integer" }
  }
  translate {
    source => "[request][headers][host]"
    target => "[service][environment]"
    dictionary_path => "/etc/logstash/host_environment_map.yml"
    refresh_interval => 300
  }
}

translate with a YAML dictionary is generally a better fit than a growing if/else chain for host→environment/region tagging — new hosts can be added by editing the file, no pipeline reload needed.

3. Local & In-Cluster Config Testing¶

logstash -f $PWD/config/logstash_processing.conf \
  --path.settings $PWD/config/ \
  --config.debug \
  --config.test_and_exit

• --config.test_and_exit — parse and exit, no ingestion. Always run before deploying.
• --config.debug — dumps the fully compiled config.
• --path.settings is separate from -f/--path.config, so you can run a local logstash.yml (e.g. dead_letter_queue.enable: true) without touching prod config.

In a pod, run the identical check against the real image:

kubectl -n <namespace> exec -it <pod> -- \
  logstash -f /run/config/logstash_processing.conf \
  --path.settings /run/config \
  --config.test_and_exit

4. Dead Letter Queue Workflow¶

Enable & pull:

dead_letter_queue.enable: true

kubectl cp -n <namespace> -c logstash-processing \
  <pod-name>:/usr/share/logstash/data/dead_letter_queue ./dlq

Read locally (read-only, repeatable):

input {
  dead_letter_queue {
    path => "/home/<user>/dlq/<pipeline-id>"
    commit_offsets => false   # don't advance the pointer — keep re-readable
  }
}
output {
  file { path => "/tmp/dlq_messages.json"; codec => json_lines }
}

Analyze:

cat /tmp/dlq_messages.json \
  | jq 'select(.message != "Event previously submitted to dead letter queue. Skipping...")' \
  | jq -s 'length'

Equivalent Kibana KQL, kept alongside so the two don't drift:

not message: "Event previously submitted to dead letter queue. Skipping..." and not runtime.region: cne2

Monitor size (single pod / fleet loop):

kubectl -n <namespace> exec -it <pod> -- \
  curl -s -XGET 'localhost:9600/_node/stats/pipelines' \
  | jq '.pipelines.main.dead_letter_queue.queue_size_in_bytes'

Retention/cleanup:

• dead_letter_queue.retain.age: 30d — auto-purge by age, measured from entry time.
• clean_consumed => true on a reprocessing dead_letter_queue input — drains segments as they're read (opposite of the read-only commit_offsets => false setup above).
• Manual flush: stop pipeline → rm -rf the DLQ path's *.log/*.log.tmp → restart. No built-in "clear" command exists — copy out via kubectl cp first if the data might matter later.

5. Environment/Region Tagging & Local-File Swap Trick¶

if [request][headers][host] == "dev.domain.tld" {
  mutate { add_field => { "[service][environment]" => "dev" } }
} else if [request][headers][host] in ["test.domain.tld1", "test.domain.tld2"] {
  mutate { add_field => { "[service][environment]" => "test" } }
}

For pointing a config at a locally-edited ruby script (e.g. masker.rb) without rebuilding the image:

sed -i 's#/run/config/masker.rb#/home/<user>/.../masker.rb#g' logstash_processing.conf
# ... test ...
sed -i 's#/home/<user>/.../masker.rb#/run/config/masker.rb#g' logstash_processing.conf   # revert before commit/deploy

Check git status/git diff on that file before every commit — the revert step is the easy thing to forget.

6. Incident Investigation Runbook¶

1. Confirm DLQ growth via node stats API (§4), across all pods if pod-specific.
2. Pull the DLQ (kubectl cp).
3. Convert to JSON locally with commit_offsets => false so re-analysis doesn't require re-pulling.
4. Narrow to the incident window, strip known-noise messages via jq, cross-check against equivalent Kibana KQL.
5. Classify remaining messages by error type (transient infra vs. genuine schema/mapping problem).
6. Reprocess (clean_consumed => true input) if fixed, or archive-then-flush if not recoverable.

7. Troubleshooting — Common Errors & Root Causes¶

Sourced from Elastic's official troubleshooting docs and the wider community.

Elasticsearch connection failures¶

Symptom: NoConnectionAvailableError / HostUnreachableError with messages like "no living connections in the connection pool." This means Logstash cannot reach Elasticsearch at all — check the ES cluster is actually up, the hosts => value is correct, and there's no network/firewall block between the two. Logstash will keep retrying with a backoff (will_retry_in_seconds) rather than dropping events, so a transient blip usually self-heals once connectivity returns.

HTTP 429 from Elasticsearch¶

A 429 means Elasticsearch's bulk/ingest queue is full and it's asking Logstash to slow down — Logstash retries automatically. Sustained 429s point at an ES-side capacity problem (undersized cluster, hot node, or too many concurrent bulk requests), not a Logstash bug. Check ES node stats for the actual bottleneck before touching Logstash's retry/backoff settings.

/tmp mounted noexec¶

Certain plugins (Netty-based inputs like TCP, and some JRuby/FFI-backed libraries) copy executable files to the temp directory at startup. If /tmp is mounted noexec, you'll see errors like FFI not available ... failed to map segment from shared object: Operation not permitted. Fix by mounting /tmp with exec, or pointing Logstash at an alternate temp dir via -Djava.io.tmpdir in jvm.options.

Config syntax errors¶

Trailing characters, missing quotes, or a bare unquoted path (e.g. path => /path/to/file.log instead of a quoted string) will fail config parsing outright. --config.test_and_exit (§3) catches these before deploy — there's no good reason to ever discover a syntax error in prod logs first.

Grok parse failures¶

Failed grok matches don't error loudly — they silently tag the event _grokparsefailure and pass it through unmodified. Periodically search Kibana for that tag to find enrichment that's silently not happening. Test patterns against real sample lines with the Grok Debugger or Grok Constructor before deploying a new pattern.

JVM heap / out-of-memory¶

Running out of heap shows up as OutOfMemoryError in the Logstash log. The immediate fix is raising -Xms/-Xmx in jvm.options, but treat that as a stopgap — repeated OOMs usually mean either a filter is buffering unexpectedly large events (check for oversized/multiline messages hitting a plugin not designed for them) or the pipeline batch size is too large for available memory.

Persistent queue backpressure / "queue is full"¶

This is not a bug — the persistent queue is deliberately designed to block input once full, protecting against data loss during a downstream slowdown, rather than silently dropping events. A persistently-full queue (refills within seconds of draining, per _node/stats/pipelines) signals a genuine steady-state rate mismatch between input and output, not a transient spike. Diagnose in this order:

1. Output destination is slow or rejecting writes — tail Logstash logs for ES 429s, Kafka producer timeouts, or S3 throttling.
2. Queue undersized for the burst profile — check _node/stats/pipelines for queue_size_in_bytes behavior over time; if it refills instantly after draining, queue.max_bytes is too small for the traffic pattern, not fundamentally broken.
3. CPU-bound filter chain (grok is the most common culprit) — check per-filter events.duration_in_millis in node stats for a hotspot.
4. Workers blocked on external lookups — the elasticsearch filter, jdbc_streaming, or DNS lookups inside the filter chain can stall workers waiting on a remote call.

The fix is always one of: speed up the output, slow down/throttle the input, or grow the queue — never treat the full queue itself as the thing to "fix."

Each input handles backpressure independently — e.g. the beats input stops accepting new connections once the persistent queue has no space, and resumes once space frees up. During a normal shutdown, in-flight events finish processing before Logstash exits; on an abnormal termination, unACKed in-flight events are reprocessed on restart, since an event is only marked ACKed once it's been handled by every configured filter and output.

Corrupt persistent queue on restart after a crash¶

If Logstash was killed mid-write to the persistent queue (e.g. OOM-killed, or queue.max_bytes hit an unsupported value on some versions), it can refuse to start with a corrupt-queue error. There's currently no clean repair path other than deleting the affected queue page files (data loss for that segment) or downgrading if the failure matches a known version-specific bug — worth checking the Logstash release notes/known-issues page for your exact version before assuming it's a one-off.

8. Performance Troubleshooting Methodology¶

The official guidance is explicit about method, not just settings: change one thing at a time, and don't start by tuning pipeline.workers. Adjusting worker count first adds a variable that makes it harder to isolate what's actually slow, so it should be a later step, not the first reflex.

Recommended order:

1. Check input/output destination performance first. Logstash can't outrun the services it talks to — an underpowered Elasticsearch cluster or a slow upstream Kafka broker caps throughput no matter how the pipeline itself is tuned.
2. Watch for disk saturation — this can also be caused indirectly, by a high error rate generating large error logs that themselves saturate disk I/O, not just by the data pipeline itself.
3. Only then touch pipeline.workers / pipeline.batch.size, one change at a time, measuring before and after each.

9. Automated / Repeatable Testing¶

Beyond --config.test_and_exit (syntax-only), two established approaches let you assert behavior — that a given input produces the expected output fields:

RSpec (via logstash-devutils)¶

Logstash's own core test suite is built this way, and it's usable for custom pipeline configs too:

require "logstash/devutils/rspec/spec_helper"

describe "Nginx filter" do
  config(File.read("conf/11-nginx-filter.conf"))

  message = '172.17.0.1 - - [05/Sep/2016:20:06:17 +0000] "GET /images/logo.png HTTP/1.1" 200 5432 "-" "-"'

  sample("message" => message, "type" => "nginx") do
    insist { subject.get("verb") } == "GET"
    insist { subject.get("response") } == 200
  end
end

Requires the logstash-devutils development gem installed via logstash-plugin install --development. This is the heavier-weight but most "native" option, and is what Elastic's own core filter/input/output plugin tests use.

logstash-filter-verifier¶

A purpose-built external tool for testing filter configs against JSON test-case files, without needing a Ruby/RSpec environment:

• Replaces a pipeline's input/output plugins so filters run in isolation against fixture input.
• Supports full pipelines.yml-based multi-pipeline setups, including pipeline-to-pipeline configs.
• Can assert against [@metadata] fields, which are otherwise invisible in normal output.
• Runs in a "daemon mode" against a directory of test cases, useful for CI.

Generally considered the easier install/onboarding path compared to RSpec if you don't already have a Ruby toolchain around, and is the community's most commonly recommended option for this specific need.

Practical CI shape¶

A reasonable pattern combining what's in this doc:

# 1. Syntax check (fast, always run)
logstash --config.test_and_exit -f conf/

# 2. Behavior check (filter-verifier or rspec, run in CI on PR)
logstash-filter-verifier daemon run --pipeline pipelines.yml --testcase-dir test/

# 3. Local manual smoke test against a copied DLQ sample, when investigating a live issue (§4)

Keeping syntax checks in a pre-commit hook and behavior tests in CI catches most of what currently relies on manual --config.test_and_exit runs plus tribal knowledge of what "looks right" in Kibana after deploy.

10. Reference¶

Your original sources (DLQ/K8s/host-tagging notes):
- Logstash settings file
- Dead letter queues — auto clean
- Node stats API
- Grok Constructor

New — official troubleshooting & performance:
- Troubleshoot Logstash — Elastic Docs
- Performance troubleshooting — Logstash
- Persistent queues — Logstash
- Logstash known issues

New — testing:
- Verifying Logstash Functionality Through Testing — Elastic Blog
- logstash-filter-verifier (GitHub)
- Testing Logstash configuration with RSpec (worked example)