Create an index template and data stream

A data stream is an append-only alias over a series of hidden backing indices (.ds-<name>-<date>-000001, -000002, …) that roll over automatically. It's the right shape for logs and metrics. A data stream can only be created when a matching index template with "data_stream": {} exists.

Naming¶

Use the <type>-<dataset>-<namespace> scheme, for example logs-opnsense-default. The built-in logs-*-* template has priority 100, so a custom template for your own dataset needs a higher priority to win.

Create the index template¶

PUT _index_template/logs-opnsense
{
  "index_patterns": ["logs-opnsense-*"],
  "data_stream": {},
  "priority": 200,
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0,
      "index.default_pipeline": "opnsense",
      "index.lifecycle.name": "logs"
    },
    "mappings": {
      "dynamic": false,
      "properties": {
        "@timestamp":        { "type": "date" },
        "message":           { "type": "match_only_text" },
        "observer":          { "properties": {
          "hostname":        { "type": "keyword" },
          "type":            { "type": "keyword" },
          "ingress":         { "properties": { "interface": { "properties": { "name": { "type": "keyword" } } } } }
        } },
        "process":           { "properties": { "name": { "type": "keyword" }, "pid": { "type": "long" } } },
        "log":               { "properties": { "syslog": { "properties": { "severity": { "properties": { "name": { "type": "keyword" } } } } } } },
        "source":            { "properties": { "ip": { "type": "ip" }, "port": { "type": "long" } } },
        "destination":       { "properties": { "ip": { "type": "ip" }, "port": { "type": "long" } } },
        "network":           { "properties": { "transport": { "type": "keyword" } } },
        "event":             { "properties": { "action": { "type": "keyword" } } },
        "error":             { "properties": { "message": { "type": "match_only_text" } } }
      }
    }
  }
}

Choices worth knowing about:

number_of_replicas: 0 suits a single-node cluster. With 1, every index stays yellow because the replica has nowhere to go. Use 1 or more on multi-node clusters.
index.default_pipeline connects the template to the ingest pipeline, so shippers don't have to name it.
index.lifecycle.name attaches an ILM policy. Without one, the data stream rolls over and grows forever.
"dynamic": false keeps unexpected fields in _source without mapping them, which prevents field explosion (see disk space).

Preview the result¶

This shows the merged settings and mappings a new backing index would get, without creating anything:

POST _index_template/_simulate_index/logs-opnsense-default

Create the data stream¶

Indexing the first document creates it automatically. To create it explicitly:

PUT _data_stream/logs-opnsense-default
GET _data_stream/logs-opnsense-default

Changing mappings later¶

Template changes only apply to new backing indices. To apply them now, force a rollover:

POST logs-opnsense-default/_rollover