Splitting Docker Stdout and Stderr With Fluentd Fluent-plugin-rewrite-tag-filter Plugin

Splitting Docker stdout and stderr logs using Fluentd and the fluent-plugin-rewrite-tag-filter plugin involves routing logs based on their stream type (stdout or stderr). This approach allows you to handle standard output and error output differently, such as sending them to separate destinations for better log management.

Overview of the Setup

1. Fluentd Input: Collect logs from Docker using the Fluentd logging driver or directly using the Fluentd forward input.
2. Filter with Rewrite Tag: Use the rewrite-tag-filter plugin to create new tags based on the stream type (stdout or stderr).
3. Match: Direct logs to different destinations based on the new tags.

Step-by-Step Configuration

1. Fluentd Configuration (fluent.conf)

Below is a Fluentd configuration that demonstrates how to achieve this setup.

Input Section: Collect Docker Logs

Set up Fluentd to receive logs from Docker containers. You can use the forward input if you're using the Fluentd logging driver in Docker.

<source>
  @type forward
  port 24224
  tag docker.*
</source>

2. Filter Section: Split Logs Using rewrite-tag-filter

Use the rewrite-tag-filter plugin to rewrite the tags based on whether the log entry is from stdout or stderr. This plugin allows you to add conditions that modify the tags dynamically.

<filter docker.**>
  @type rewrite_tag_filter
  # Match based on the log stream type (stdout or stderr)
  <rule>
    key stream
    pattern stdout
    tag stdout.logs
  </rule>
  <rule>
    key stream
    pattern stderr
    tag stderr.logs
  </rule>
</filter>

3. Match Section: Route Logs Based on the Rewritten Tags

Once the tags have been rewritten, you can match these new tags (stdout.logs and stderr.logs) to route the logs to different outputs.

# Match for stdout logs
<match stdout.logs>
  @type file
  path /var/log/fluentd/stdout_logs.log
  <buffer>
    @type file
    path /var/log/fluentd/buffer/stdout
  </buffer>
  <format>
    @type json
  </format>
</match>

# Match for stderr logs
<match stderr.logs>
  @type file
  path /var/log/fluentd/stderr_logs.log
  <buffer>
    @type file
    path /var/log/fluentd/buffer/stderr
  </buffer>
  <format>
    @type json
  </format>
</match>

Explanation of the Configuration

1. Source:
   • The <source> section uses the forward plugin to collect logs from Docker containers tagged with docker.*.
2. Filter:
   • The <filter> section uses the rewrite_tag_filter plugin to inspect the stream field, which indicates whether the log is from stdout or stderr.
   • It then rewrites the tag based on the stream type:
     • stdout logs are tagged as stdout.logs.
     • stderr logs are tagged as stderr.logs.
3. Match:
   • The <match> sections specify where to send the rewritten logs.
   • In this example, stdout logs are sent to /var/log/fluentd/stdout_logs.log and stderr logs to /var/log/fluentd/stderr_logs.log.

Installing fluent-plugin-rewrite-tag-filter

Ensure that the rewrite-tag-filter plugin is installed in your Fluentd environment. You can install it using the following command:

td-agent-gem install fluent-plugin-rewrite-tag-filter

Testing the Configuration

1. Restart Fluentd to apply the new configuration:

    

   Copied!

   sudo systemctl restart td-agent
   

2. Check Logs: Verify that the logs are being split correctly into their respective files by checking /var/log/fluentd/stdout_logs.log and /var/log/fluentd/stderr_logs.log.

Additional Considerations

• Buffering: Ensure proper buffering is configured to handle high log volumes and prevent data loss in case of service interruptions.
• Performance: Test the setup under load conditions similar to production to ensure Fluentd handles log processing efficiently.
• Security: Set appropriate file permissions on log files and buffers to prevent unauthorized access.

Got an article suggestion? Let us know

Explore more

Fluentd

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.