# Splitting Docker Stdout and Stderr With Fluentd Fluent-plugin-rewrite-tag-filter Plugin

Splitting Docker `stdout` and `stderr` logs using Fluentd and the `fluent-plugin-rewrite-tag-filter` plugin involves routing logs based on their stream type (`stdout` or `stderr`). This approach allows you to handle standard output and error output differently, such as sending them to separate destinations for better log management.

### **Overview of the Setup**

1. **Fluentd Input**: Collect logs from Docker using the Fluentd logging driver or directly using the Fluentd forward input.
2. **Filter with Rewrite Tag**: Use the `rewrite-tag-filter` plugin to create new tags based on the stream type (`stdout` or `stderr`).
3. **Match**: Direct logs to different destinations based on the new tags.

### **Step-by-Step Configuration**

### **1. Fluentd Configuration (`fluent.conf`)**

Below is a Fluentd configuration that demonstrates how to achieve this setup.

### **Input Section: Collect Docker Logs**

Set up Fluentd to receive logs from Docker containers. You can use the `forward` input if you're using the Fluentd logging driver in Docker.

```
<source>
  @type forward
  port 24224
  tag docker.*
</source>
```

### **2. Filter Section: Split Logs Using `rewrite-tag-filter`**

Use the `rewrite-tag-filter` plugin to rewrite the tags based on whether the log entry is from `stdout` or `stderr`. This plugin allows you to add conditions that modify the tags dynamically.

```
<filter docker.**>
  @type rewrite_tag_filter
  # Match based on the log stream type (stdout or stderr)
  <rule>
    key stream
    pattern stdout
    tag stdout.logs
  </rule>
  <rule>
    key stream
    pattern stderr
    tag stderr.logs
  </rule>
</filter>

```

### **3. Match Section: Route Logs Based on the Rewritten Tags**

Once the tags have been rewritten, you can match these new tags (`stdout.logs` and `stderr.logs`) to route the logs to different outputs.

```
# Match for stdout logs
<match stdout.logs>
  @type file
  path /var/log/fluentd/stdout_logs.log
  <buffer>
    @type file
    path /var/log/fluentd/buffer/stdout
  </buffer>
  <format>
    @type json
  </format>
</match>

# Match for stderr logs
<match stderr.logs>
  @type file
  path /var/log/fluentd/stderr_logs.log
  <buffer>
    @type file
    path /var/log/fluentd/buffer/stderr
  </buffer>
  <format>
    @type json
  </format>
</match>

```

### **Explanation of the Configuration**

1. **Source**:
    - The `<source>` section uses the `forward` plugin to collect logs from Docker containers tagged with `docker.*`.
2. **Filter**:
    - The `<filter>` section uses the `rewrite_tag_filter` plugin to inspect the `stream` field, which indicates whether the log is from `stdout` or `stderr`.
    - It then rewrites the tag based on the stream type:
        - `stdout` logs are tagged as `stdout.logs`.
        - `stderr` logs are tagged as `stderr.logs`.
3. **Match**:
    - The `<match>` sections specify where to send the rewritten logs.
    - In this example, `stdout` logs are sent to `/var/log/fluentd/stdout_logs.log` and `stderr` logs to `/var/log/fluentd/stderr_logs.log`.

### **Installing `fluent-plugin-rewrite-tag-filter`**

Ensure that the `rewrite-tag-filter` plugin is installed in your Fluentd environment. You can install it using the following command:

```bash
td-agent-gem install fluent-plugin-rewrite-tag-filter
```

### **Testing the Configuration**

1. **Restart Fluentd** to apply the new configuration:
    
    ```bash
    sudo systemctl restart td-agent
    ```
    
2. **Check Logs**: Verify that the logs are being split correctly into their respective files by checking `/var/log/fluentd/stdout_logs.log` and `/var/log/fluentd/stderr_logs.log`.

### **Additional Considerations**

- **Buffering**: Ensure proper buffering is configured to handle high log volumes and prevent data loss in case of service interruptions.
- **Performance**: Test the setup under load conditions similar to production to ensure Fluentd handles log processing efficiently.
- **Security**: Set appropriate file permissions on log files and buffers to prevent unauthorized access.