Fluent-bit - Splitting Json Log Into Structured Fields in Elasticsearch

To split JSON logs into structured fields in Elasticsearch using Fluent Bit, you need to properly configure Fluent Bit to parse the JSON log data and then send it in a structured format to Elasticsearch. Fluent Bit is a lightweight log processor and forwarder often used to collect data before sending it to data sinks like Elasticsearch.

Here's a step-by-step guide on how to achieve this:

Step 1: Set Up Fluent Bit Input

Configure the Fluent Bit input section to read your JSON logs. For example, if you are reading logs from a file, you can use the tail input plugin:

[INPUT]
    Name        tail
    Path        /path/to/your/log/file.log
    Parser      json_parser
    Tag         app.logs

• Name: Specifies the input plugin (tail for reading log files).
• Path: The path to your log file.
• Parser: Defines which parser to use (we'll define this in the next step).
• Tag: A tag to identify this log stream.

Step 2: Configure the JSON Parser

Define a JSON parser in the Fluent Bit configuration file that parses the incoming log lines into structured JSON objects.

[PARSER]
    Name        json_parser
    Format      json
    Time_Key    timestamp
    Time_Format %Y-%m-%dT%H:%M:%S.%LZ

• Name: Name of the parser.
• Format: Set to json to parse incoming log lines as JSON.
• Time_Key and Time_Format: Configure the timestamp parsing if your logs have a timestamp field.

Step 3: Set Up Fluent Bit Filter (Optional)

If you need to modify, rename, or drop certain fields, you can use the modify or record_modifier filter plugins. Here is an example of a filter configuration:

[FILTER]
    Name        modify
    Match       app.logs
    Rename      old_field_name  new_field_name
    Remove_key  field_to_remove

• Match: Specifies which logs to apply the filter to (based on the tag).
• Rename: Rename fields if needed.
• Remove_key: Remove unnecessary fields.

Step 4: Set Up the Output to Elasticsearch

Configure the Fluent Bit output plugin to send the structured logs to Elasticsearch:

[OUTPUT]
    Name            es
    Match           app.logs
    Host            your_elasticsearch_host
    Port            9200
    Index           your_index_name
    Type            _doc
    Logstash_Format On
    Replace_Dots    On

• Name: The output plugin, set to es for Elasticsearch.
• Match: Defines which logs to send to this output based on the tag.
• Host and Port: Your Elasticsearch host and port.
• Index: The name of the index to which logs are sent.
• Replace_Dots: Replaces dots in field names, as Elasticsearch doesn’t allow dots in field names by default.

Step 5: Run Fluent Bit

Once your configuration file is set up, run Fluent Bit with the configuration file:

fluent-bit -c /path/to/your/fluent-bit.conf

Expected Result

With this setup, Fluent Bit will:

1. Read the log entries from the specified path.
2. Parse the JSON log entries into structured fields using the defined parser.
3. Optionally modify the structure using filters.
4. Send the structured data to Elasticsearch, where each JSON field will become an individual, searchable field in the index.

Got an article suggestion? Let us know

Explore more

FluentBit

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.