How to Run Fluentd in Docker Within the Internal Network

Running Fluentd in Docker within an internal network is a common setup for aggregating and processing logs in containerized environments. To achieve this, you need to configure Docker networking so Fluentd can communicate with other containers and services within your internal network.

Step-by-Step Guide to Running Fluentd in Docker Within an Internal Network

1. Create a Docker Network

First, create a Docker network for internal communication:

docker network create fluentd-network

This creates a network named fluentd-network that you can use to connect containers.

2. Create a Fluentd Configuration File

Prepare a Fluentd configuration file (fluentd.conf) with your desired settings. For example:

<source>
  @type forward
  port 24224
  tag docker.*
</source>

<filter docker.**>
  @type record_transformer
  <record>
    hostname ${hostname}
    environment production
  </record>
</filter>

<match docker.**>
  @type stdout
</match>

This configuration listens for logs on port 24224 and prints them to the standard output.

3. Create a Dockerfile for Fluentd

If you want to build a custom Docker image for Fluentd, create a Dockerfile:

FROM fluent/fluentd:v1.14-1

USER root
COPY fluentd.conf /fluentd/etc/fluentd.conf

RUN gem install fluent-plugin-record-transformer
RUN gem install fluent-plugin-rewrite-tag-filter

USER fluent

This Dockerfile uses the Fluentd base image, copies your configuration file into the container, and installs necessary plugins.

4. Build the Docker Image

Build the Fluentd Docker image:

docker build -t my-fluentd-image .

5. Run Fluentd Container

Run the Fluentd container on the internal network:

docker run -d --name fluentd \\
  --network fluentd-network \\
  -p 24224:24224 \\
  -v /path/to/fluentd.conf:/fluentd/etc/fluentd.conf \\
  my-fluentd-image

6. Run Other Containers on the Same Network

Ensure other containers that need to send logs to Fluentd are also connected to the same network.

For example, to run a container with Fluentd as the logging driver:

docker run -d --name my-app \\
  --network fluentd-network \\
  --log-driver=fluentd \\
  --log-opt fluentd-address=fluentd:24224 \\
  my-app-image

7. Verify Connectivity

To ensure that Fluentd and other containers can communicate:

1. Check Fluentd Logs: Verify that Fluentd is receiving logs:

    

   Copied!

   docker logs fluentd
   

2. Check Container Logs: Ensure the application container is sending logs:

    

   Copied!

   docker logs my-app
   

3. Network Inspection: Inspect the Docker network to verify connected containers:

    

   Copied!

   docker network inspect fluentd-network
   

8. Access Fluentd from Other Services

If you have services outside of Docker containers that need to send logs to Fluentd, ensure they are configured to send logs to the host's IP address or domain name on the Fluentd port (e.g., localhost:24224).

9. Example of Using Fluentd with Docker Compose

For a more integrated setup, you can use Docker Compose to manage Fluentd and other services.

docker-compose.yml

version: '3'
services:
  fluentd:
    image: my-fluentd-image
    networks:
      - fluentd-network
    ports:
      - "24224:24224"
    volumes:
      - ./fluentd.conf:/fluentd/etc/fluentd.conf

  my-app:
    image: my-app-image
    networks:
      - fluentd-network
    logging:
      driver: "fluentd"
      options:
        fluentd-address: "fluentd:24224"

networks:
  fluentd-network:
    driver: bridge

In this setup, Docker Compose handles network creation and container management.

Summary

• Create a Docker Network: Establish a network for internal communication.
• Configure Fluentd: Set up Fluentd configuration and Dockerfile.
• Run Containers: Start Fluentd and other containers on the same network.
• Verify: Ensure containers can communicate and logs are processed correctly.
• Optional: Use Docker Compose for a simplified setup.

Got an article suggestion? Let us know

Explore more

Fluentd

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.