List of Syntax for Logstash's Grok

Logstash's Grok filter provides predefined patterns for parsing logs and extracting fields. Here’s a guide to common Grok syntax patterns and examples for their usage.

Common Grok Patterns

1. Data Types

   • %{WORD}: Matches a single word, useful for extracting non-space text.
   • %{NUMBER}: Matches an integer or decimal.
   • %{INT}: Matches an integer (no decimals).
   • %{FLOAT}: Matches a floating-point number.

   Example:

    

   Copied!

   "User ID: %{NUMBER:user_id}"
   

2. Date and Time

   • %{TIMESTAMP_ISO8601}: Matches an ISO 8601 timestamp (yyyy-MM-dd'T'HH:mm:ss.SSSZ).
   • %{DATE_US}: Matches a date in MM/dd/yyyy format.
   • %{DATE_EU}: Matches a date in dd/MM/yyyy format.
   • %{TIME}: Matches a time in HH:mm:ss format.

   Example:

    

   Copied!

   "%{TIMESTAMP_ISO8601:timestamp}"
   

3. Network

   • %{IP}: Matches an IPv4 or IPv6 address.
   • %{HOSTNAME}: Matches a hostname.
   • %{MAC}: Matches a MAC address.

   Example:

    

   Copied!

   "Client IP: %{IP:client_ip}"
   

4. Logs and HTTP Patterns

   • %{COMMONAPACHELOG}: Parses a typical Apache log format.
   • %{COMBINEDAPACHELOG}: Parses Apache combined log format.
   • %{HTTPDATE}: Matches an HTTP-style date.

   Example:

    

   Copied!

   "%{COMBINEDAPACHELOG}"
   

5. Custom Patterns

   • %{GREEDYDATA}: Matches everything (often used at the end).
   • %{DATA}: Matches anything (less greedy than GREEDYDATA).

   Example:

    

   Copied!

   "%{WORD:loglevel} %{GREEDYDATA:message}"
   

Example Grok Filter

For an Apache log entry:

127.0.0.1 - - [25/Oct/2024:10:15:00 +0000] "GET /index.html HTTP/1.1" 200 2326

Grok filter configuration:

filter {
  grok {
    match => { "message" => '%{IP:client_ip} %{USER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] "%{WORD:method} %{URIPATH:request} HTTP/%{NUMBER:http_version}" %{NUMBER:response} %{NUMBER:bytes}' }
  }
}

This filter extracts fields like client_ip, timestamp, method, request, response, and bytes from each log entry.

Tips for Grok Pattern Usage

• Combine patterns to match complex log structures.
• Use ? for optional fields, such as "%{USER:user}?".
• Test patterns with the Grok Debugger in Kibana or online tools to validate before deploying to production.

These patterns and tips should help you parse various types of logs efficiently with Grok in Logstash.

Got an article suggestion? Let us know

Explore more

Logstash

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.