Can You Use Environment Variables in Config File for Fluentd

Yes, you can use environment variables in Fluentd configuration files to make them more flexible and manageable, especially for deployments that require dynamic configuration or secret management. Fluentd supports environment variable substitution directly in its configuration.

Using Environment Variables in Fluentd Configuration

1. Basic Example

You can reference environment variables in the Fluentd configuration file using the ${ENV_VAR} syntax. Here’s a basic example:

<source>
  @type forward
  port ${FLUENTD_PORT}
  tag ${FLUENTD_TAG}
</source>

<match ${FLUENTD_TAG}>
  @type file
  path /var/log/${FLUENTD_LOG_DIR}/fluentd.log
  <format>
    @type json
  </format>
</match>

In this example:

• FLUENTD_PORT, FLUENTD_TAG, and FLUENTD_LOG_DIR are environment variables that you need to set in your environment before running Fluentd.

2. Setting Environment Variables

You can set environment variables in your shell or within a Docker container. For example:

In a Shell:

export FLUENTD_PORT=24224
export FLUENTD_TAG=app.logs
export FLUENTD_LOG_DIR=logs

In Docker:

You can set environment variables in the Dockerfile or using the docker run command:

# Dockerfile example
ENV FLUENTD_PORT=24224
ENV FLUENTD_TAG=app.logs
ENV FLUENTD_LOG_DIR=logs

# Docker run example
docker run -d --name fluentd \\
  -e FLUENTD_PORT=24224 \\
  -e FLUENTD_TAG=app.logs \\
  -e FLUENTD_LOG_DIR=logs \\
  fluent/fluentd

3. Using Environment Variables in Kubernetes

If you’re running Fluentd in Kubernetes, you can set environment variables in your Pod specification:

apiVersion: v1
kind: Pod
metadata:
  name: fluentd
spec:
  containers:
    - name: fluentd
      image: fluent/fluentd
      env:
        - name: FLUENTD_PORT
          value: "24224"
        - name: FLUENTD_TAG
          value: "app.logs"
        - name: FLUENTD_LOG_DIR
          value: "logs"
      volumeMounts:
        - name: config-volume
          mountPath: /fluentd/etc
  volumes:
    - name: config-volume
      configMap:
        name: fluentd-config

In this setup, ensure that the ConfigMap named fluentd-config contains your Fluentd configuration file.

4. Advanced Usage with fluent-plugin-env

For more advanced use cases, you might want to use the fluent-plugin-env plugin, which provides more options for environment variable substitution. This plugin allows for a more complex environment variable usage:

1. Install the Plugin:

    

   Copied!

   td-agent-gem install fluent-plugin-env
   

2. Configure Fluentd to Use fluent-plugin-env:

    

   Copied!

   <source>
     @type env
   </source>
   
   <match ${FLUENTD_TAG}>
     @type file
     path /var/log/${FLUENTD_LOG_DIR}/fluentd.log
     <format>
       @type json
     </format>
   </match>
   

5. Testing and Validation

1. Set Environment Variables: Ensure environment variables are correctly set in your environment or container.
2. Restart Fluentd: Restart Fluentd to apply the new configuration and environment variables.
3. Verify Logs: Check the logs or outputs to confirm that Fluentd is using the environment variables as expected.

Using environment variables in Fluentd configuration enhances flexibility and simplifies configuration management, especially when deploying across different environments or scaling applications.

Got an article suggestion? Let us know

Explore more

Fluentd

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.