What Is the Format of Logstash Config File

The Logstash configuration file (.conf) is structured to define how Logstash processes and transforms data. It consists of three main sections: input, filter, and output. Each section is responsible for a different stage of the data pipeline.

Basic Structure of Logstash Configuration File

input {
  # input plugins configuration
}

filter {
  # filter plugins configuration (optional)
}

output {
  # output plugins configuration
}

1. Input Section

The input section specifies the source(s) from which Logstash will receive the data. You can use multiple input plugins like file, tcp, http, etc.

Example (input from a file):

input {
  file {
    path => "/var/log/syslog"
    start_position => "beginning"
  }
}

• path: Specifies the file path to read data from.
• start_position: Defines whether to start reading the file from the beginning or from the end (for newly appended logs).

2. Filter Section

The filter section is used for data transformation and parsing. Common filters include grok, mutate, date, and geoip. Filters allow you to manipulate, parse, and enrich the data before sending it to the output.

Example (filter using Grok):

filter {
  grok {
    match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{WORD:logsource} %{GREEDYDATA:logmessage}" }
  }

  date {
    match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    target => "@timestamp"
  }

  mutate {
    remove_field => [ "timestamp" ]
  }
}

• grok: Used for pattern matching to parse unstructured log data into structured data.
• date: Used to parse date fields and set the @timestamp field to match the parsed time.
• mutate: Can be used to modify fields, such as renaming, removing, or converting field types.

3. Output Section

The output section defines where the data should be sent. Outputs can include destinations like Elasticsearch, a file, a database, or another system.

Example (output to Elasticsearch):

output {
  elasticsearch {
    hosts => ["<http://localhost:9200>"]
    index => "logstash-%{+YYYY.MM.dd}"
  }

  stdout { codec => rubydebug }  # Output to console for debugging
}

• hosts: Specifies the address of your Elasticsearch instance.
• index: Defines the index pattern for storing the logs in Elasticsearch (with dynamic date-based indices).
• stdout: Sends output to the console, using the rubydebug codec to make it human-readable (useful for debugging).

Example of a Complete Logstash Configuration File

input {
  file {
    path => "/var/log/syslog"
    start_position => "beginning"
  }
}

filter {
  grok {
    match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{WORD:logsource} %{GREEDYDATA:logmessage}" }
  }

  date {
    match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    target => "@timestamp"
  }

  mutate {
    remove_field => [ "timestamp" ]
  }
}

output {
  elasticsearch {
    hosts => ["<http://localhost:9200>"]
    index => "logstash-%{+YYYY.MM.dd}"
  }

  stdout { codec => rubydebug }
}

Additional Notes:

• Multiple Inputs, Filters, and Outputs: Logstash allows multiple input, filter, and output blocks within a single configuration file. Each block will be processed independently.

• Conditionals: You can use conditionals to apply specific filters or outputs based on certain criteria:

   

  Copied!

  filter {
    if [logsource] == "apache" {
      grok { ... }
    }
  }
  

Conclusion

The Logstash configuration file is highly flexible and modular, allowing you to specify multiple data sources, apply complex transformations, and send data to various outputs. The input, filter, and output sections are the core elements that define how Logstash processes data from start to finish.

Got an article suggestion? Let us know

Explore more

Logstash

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.