# Docker Fluentd Logging Driver for Multiline

The Docker Fluentd logging driver is often used to send container logs to Fluentd, which can then forward logs to various destinations like Elasticsearch, Splunk, or other data stores. Handling multiline logs, such as stack traces, requires some additional configuration because Docker's default logging driver does not handle multiline logs well.

### **Setting Up Docker Fluentd Logging Driver for Multiline Logs**

To handle multiline logs effectively, you need to configure Fluentd to properly parse and process these logs using a parser that can handle multiline patterns, such as `multiline` or `regexp`. Below are the steps to set up Docker with Fluentd to handle multiline logs.

### **Step 1: Configure Docker to Use Fluentd Logging Driver**

Configure your Docker containers to use the Fluentd logging driver. You can set this up in the Docker daemon configuration (`/etc/docker/daemon.json`) or specify it in the container run command.

### **Configure Docker Daemon (Globally)**

Update the Docker daemon configuration file (`/etc/docker/daemon.json`) to use Fluentd as the logging driver:

```json
{
  "log-driver": "fluentd",
  "log-opts": {
    "fluentd-address": "localhost:24224",
    "tag": "docker.{{.Name}}"
  }
}

```

### **Specify Fluentd in `docker run` Command (Per Container)**

Alternatively, specify the Fluentd logging driver when running a container:

```bash
docker run --log-driver=fluentd --log-opt tag="docker.{{.Name}}" your-image
```

### **Step 2: Set Up Fluentd to Handle Multiline Logs**

Fluentd requires a proper configuration to handle multiline logs. You need to use a parser that can merge multiple log lines into a single event based on patterns.

### **Fluentd Configuration Example (`fluent.conf`)**

Below is an example Fluentd configuration file that handles multiline logs. It uses the `tail` input plugin with a `multiline` parser to combine logs that match a specific pattern.

```
<source>
  @type forward
  port 24224
</source>

# Multiline Parser Configuration
<filter docker.**>
  @type parser
  key_name log
  <parse>
    @type multiline
    format_firstline /^(?<time>[^ ]* [^ ]*) (?<severity>[A-Z]+) (?<message>.*)/
    format1 /^(?<time>[^ ]* [^ ]*) (?<severity>[A-Z]+) (?<message>.*)/
    multiline_flush_interval 5
  </parse>
</filter>

<match docker.**>
  @type stdout
</match>

```

### **Explanation of the Configuration**

- **`@type forward`**: The input plugin listens for logs from Docker on port 24224.
- **`@type parser`**: The filter applies a parser to the incoming log stream.
- **`key_name log`**: Specifies the key where the log message is stored.
- **`<parse>` section**:
    - **`@type multiline`**: Uses the multiline parser to handle multiline log entries.
    - **`format_firstline`**: Defines the pattern that identifies the first line of a log entry.
    - **`format1`**: Defines the pattern for additional lines of the log entry.
    - **`multiline_flush_interval`**: Sets how often Fluentd should flush incomplete multiline logs (in seconds).

### **Step 3: Restart Fluentd and Docker**

1. **Restart Fluentd**:
    
    ```bash
    sudo systemctl restart fluentd
    ```
    
2. **Restart Docker**:
    
    ```bash
    sudo systemctl restart docker
    ```
    

### **Additional Considerations**

- **Pattern Adjustments**: The `format_firstline` and `format1` regex patterns should be adjusted based on the specific structure of your logs, especially if dealing with stack traces or other specific formats.
- **Performance Impact**: Multiline parsing can be CPU-intensive. Be mindful of performance impacts, especially when handling a high volume of logs.
- **Testing**: Always test your configuration with sample logs to ensure the multiline parsing behaves as expected.