How to Create Multiple Indexes in Logstash.conf File?

Creating multiple indexes in a Logstash configuration file allows you to organize your data efficiently based on different criteria or log sources. This can be achieved by defining multiple output sections within your configuration file. Below is a step-by-step guide on how to set this up.

Step-by-Step Guide to Create Multiple Indexes

Step 1: Install Logstash

If you haven't already installed Logstash, refer to the official installation guide for your operating system.

Step 2: Prepare Your Logstash Configuration File

Create or edit a Logstash configuration file (e.g., logstash.conf) to define your inputs, filters, and outputs. Here’s an example configuration that creates multiple indexes based on different log types.

Example Logstash Configuration

input {
  file {
    path => "/path/to/your/logs/*.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"  # Use for testing; avoid storing positions
  }
}

filter {
  # Example filter to distinguish log types
  if [type] == "app_log" {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:log_message}" }
    }
    mutate {
      add_field => { "index_name" => "app-logs" }
    }
  } else if [type] == "error_log" {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} ERROR: %{GREEDYDATA:error_message}" }
    }
    mutate {
      add_field => { "index_name" => "error-logs" }
    }
  }
}

output {
  if [index_name] == "app-logs" {
    elasticsearch {
      hosts => ["<http://localhost:9200>"]
      index => "app-logs-%{+YYYY.MM.dd}"  # Daily index for app logs
    }
  } else if [index_name] == "error-logs" {
    elasticsearch {
      hosts => ["<http://localhost:9200>"]
      index => "error-logs-%{+YYYY.MM.dd}"  # Daily index for error logs
    }
  }
}

Explanation of the Configuration

1. Input Section:
   • The file input plugin reads logs from a specified directory. The sincedb_path option is set to /dev/null for testing, meaning it won’t remember the last read position.
2. Filter Section:
   • The filter block checks the type of log (defined elsewhere, such as through a field in the log message) and processes it accordingly.
   • It uses grok to parse the log message and extract relevant fields.
   • It then adds an index_name field to differentiate between log types.
3. Output Section:
   • Based on the value of index_name, different Elasticsearch outputs are defined.
   • Each output specifies a unique index name format, allowing you to create daily indexes for both application logs and error logs.

Step 3: Run Logstash

To start Logstash with your configuration, use the following command:

bin/logstash -f /path/to/your/logstash.conf

Conclusion

By structuring your Logstash configuration file to include multiple output conditions, you can easily create and manage multiple indexes in Elasticsearch. This method allows you to organize your data based on log types, enabling better data management and retrieval. Make sure to adjust the grok patterns and filters according to your specific log formats for optimal results.

Got an article suggestion? Let us know

Explore more

Logstash

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.