Docker Fluentd Logging Driver for Multiline

The Docker Fluentd logging driver is often used to send container logs to Fluentd, which can then forward logs to various destinations like Elasticsearch, Splunk, or other data stores. Handling multiline logs, such as stack traces, requires some additional configuration because Docker's default logging driver does not handle multiline logs well.

Setting Up Docker Fluentd Logging Driver for Multiline Logs

To handle multiline logs effectively, you need to configure Fluentd to properly parse and process these logs using a parser that can handle multiline patterns, such as multiline or regexp. Below are the steps to set up Docker with Fluentd to handle multiline logs.

Step 1: Configure Docker to Use Fluentd Logging Driver

Configure your Docker containers to use the Fluentd logging driver. You can set this up in the Docker daemon configuration (/etc/docker/daemon.json) or specify it in the container run command.

Configure Docker Daemon (Globally)

Update the Docker daemon configuration file (/etc/docker/daemon.json) to use Fluentd as the logging driver:

{
  "log-driver": "fluentd",
  "log-opts": {
    "fluentd-address": "localhost:24224",
    "tag": "docker.{{.Name}}"
  }
}

Specify Fluentd in docker run Command (Per Container)

Alternatively, specify the Fluentd logging driver when running a container:

docker run --log-driver=fluentd --log-opt tag="docker.{{.Name}}" your-image

Step 2: Set Up Fluentd to Handle Multiline Logs

Fluentd requires a proper configuration to handle multiline logs. You need to use a parser that can merge multiple log lines into a single event based on patterns.

Fluentd Configuration Example (fluent.conf)

Below is an example Fluentd configuration file that handles multiline logs. It uses the tail input plugin with a multiline parser to combine logs that match a specific pattern.

<source>
  @type forward
  port 24224
</source>

# Multiline Parser Configuration
<filter docker.**>
  @type parser
  key_name log
  <parse>
    @type multiline
    format_firstline /^(?<time>[^ ]* [^ ]*) (?<severity>[A-Z]+) (?<message>.*)/
    format1 /^(?<time>[^ ]* [^ ]*) (?<severity>[A-Z]+) (?<message>.*)/
    multiline_flush_interval 5
  </parse>
</filter>

<match docker.**>
  @type stdout
</match>

Explanation of the Configuration

• @type forward: The input plugin listens for logs from Docker on port 24224.
• @type parser: The filter applies a parser to the incoming log stream.
• key_name log: Specifies the key where the log message is stored.
• <parse> section:
  • @type multiline: Uses the multiline parser to handle multiline log entries.
  • format_firstline: Defines the pattern that identifies the first line of a log entry.
  • format1: Defines the pattern for additional lines of the log entry.
  • multiline_flush_interval: Sets how often Fluentd should flush incomplete multiline logs (in seconds).

Step 3: Restart Fluentd and Docker

1. Restart Fluentd:

    

   Copied!

   sudo systemctl restart fluentd
   

2. Restart Docker:

    

   Copied!

   sudo systemctl restart docker
   

Additional Considerations

• Pattern Adjustments: The format_firstline and format1 regex patterns should be adjusted based on the specific structure of your logs, especially if dealing with stack traces or other specific formats.
• Performance Impact: Multiline parsing can be CPU-intensive. Be mindful of performance impacts, especially when handling a high volume of logs.
• Testing: Always test your configuration with sample logs to ensure the multiline parsing behaves as expected.

Got an article suggestion? Let us know

Explore more

Fluentd

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.