Logstash Date Parsing as Timestamp Using the Date Filter

In Logstash, you can use the date filter to parse dates from log messages and set them as the @timestamp field in your events. This is useful when the timestamp in your logs doesn’t match Logstash's default @timestamp format (which is the current time when the event is processed).

Basic Date Filter Configuration

1. Use the date filter: Extract the date from your log event.
2. Specify the date format: Match it to your log's date pattern.

Example Configuration

filter {
  grok {
    match => { "message" => "\\[%{TIMESTAMP_ISO8601:log_timestamp}\\] %{LOGLEVEL:loglevel} %{GREEDYDATA:log_message}" }
  }

  date {
    match => ["log_timestamp", "ISO8601"]  # Use the appropriate format for your date
    target => "@timestamp"  # Sets the parsed date as @timestamp
    timezone => "UTC"       # Set timezone if needed (default is UTC)
  }
}

Explanation

• log_timestamp: Field created by the grok filter, containing the timestamp in your log entry.
• date filter: Converts log_timestamp into the @timestamp field.
• match: Specifies the format. Use "ISO8601" for standard formats, or specify a custom format (e.g., "dd/MMM/yyyy:HH:mm:ss Z").
• timezone: Adjust if your logs are in a timezone other than UTC.

Example Date Formats

• ISO8601: Standard format ("yyyy-MM-dd'T'HH:mm:ss.SSSZ")
• Custom Format: If your log uses a different date format, specify it (e.g., "MMM dd HH:mm:ss", "yyyy-MM-dd HH:mm:ss").

Complete Example for Custom Date

For a log entry like this:

Oct 25 2024 10:15:00 ERROR Some error message

You’d use:

filter {
  grok {
    match => { "message" => "%{MONTH:month} %{MONTHDAY:day} %{YEAR:year} %{TIME:time} %{LOGLEVEL:loglevel} %{GREEDYDATA:log_message}" }
  }

  mutate {
    add_field => { "log_timestamp" => "%{month} %{day} %{year} %{time}" }
  }

  date {
    match => ["log_timestamp", "MMM dd yyyy HH:mm:ss"]
    target => "@timestamp"
  }
}

This approach allows @timestamp to align with the original log’s timestamp, making time-based analysis more accurate.

Got an article suggestion? Let us know

Explore more

Logstash

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.