Logstash Grok Multiline Message

To process and parse multiline messages with Logstash and the Grok filter, you need to:

1. Combine the multiline logs into a single event using the multiline codec.
2. Use the grok filter to extract fields from the combined multiline log entry.

Here’s how to set it up:

Step 1: Combine Multiline Logs

In the file input, use the multiline codec to group multiline messages based on a specific pattern that matches the start of a new log entry.

input {
  file {
    path => "/path/to/your/logs/*.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"

    codec => multiline {
      pattern => "^\\["                  # Example: Starts with a timestamp or specific pattern
      negate => true                    # Indicates lines not matching the pattern should be appended
      what => "previous"                # Appends lines to the previous matching line
    }
  }
}

Step 2: Parse Combined Logs with Grok

Once the multiline message is combined into a single event, use the grok filter to extract fields from it. Make sure your Grok pattern matches the structure of the multiline log entry.

filter {
  grok {
    match => { "message" => "\\[%{TIMESTAMP_ISO8601:timestamp}\\] %{LOGLEVEL:loglevel} %{GREEDYDATA:log_message}" }
  }

  # Additional processing if needed
}

Example

Consider a multiline log entry like this:

[2024-10-25 10:15:00] ERROR Something went wrong
Details: Connection timed out after 10 seconds

This configuration:

1. Multiline codec: Combines the log entry lines into a single event.
2. Grok filter: Extracts timestamp, loglevel, and log_message fields.

The log_message field will contain the complete message after the log level, including any details in the subsequent lines.

Got an article suggestion? Let us know

Explore more

Logstash

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.