Fluentd: One Source for Several Filters and Matches

In Fluentd, it's common to use a single source to collect logs and then process them through multiple filters and match patterns. This setup allows you to route and manipulate logs flexibly, applying different filters to the same source data and directing the results to various outputs.

Basic Fluentd Configuration: One Source, Multiple Filters, and Matches

Below is a step-by-step guide on how to set up a Fluentd configuration with one source and several filters and matches.

1. Define the Source

Start by defining a single source that collects logs. For example, you might use the tail input plugin to read logs from a file.

<source>
  @type tail
  path /var/log/app.log
  pos_file /var/log/td-agent/app.log.pos
  tag app.logs
  format /^(?<time>[^ ]* [^ ]*) (?<severity>[A-Z]+) (?<message>.*)/
  time_format %Y-%m-%d %H:%M:%S
</source>

2. Apply Multiple Filters

Filters allow you to modify or enrich log data before it's sent to its destination. You can use multiple filters on the same source by matching on the source's tag.

Example Filters Configuration

# Filter 1: Parse JSON fields from the log message
<filter app.logs>
  @type parser
  key_name message
  <parse>
    @type json
    # Additional parser settings can be added here
  </parse>
</filter>

# Filter 2: Add or modify fields
<filter app.logs>
  @type record_transformer
  <record>
    hostname ${hostname}
    app_name my_app
  </record>
</filter>

# Filter 3: Use grep to filter out logs that don't match a pattern
<filter app.logs>
  @type grep
  <regexp>
    key severity
    pattern WARN|ERROR
  </regexp>
</filter>

3. Define Multiple Matches

Matches define where and how the filtered logs are sent. You can have multiple matches with different tags or patterns to direct logs to various destinations.

Example Matches Configuration

# Match 1: Send logs to Elasticsearch
<match app.logs>
  @type elasticsearch
  host localhost
  port 9200
  index_name app-logs
  type_name _doc
</match>

# Match 2: Send logs to a file for archiving
<match app.logs>
  @type file
  path /var/log/fluentd/archived_logs.log
  append true
  <format>
    @type json
  </format>
</match>

# Match 3: Send critical logs to Slack for alerts
<match app.logs>
  @type slack
  webhook_url <https://hooks.slack.com/services/your/slack/webhook>
  channel #alerts
  username fluentd
  <buffer>
    @type memory
    chunk_limit_size 1m
  </buffer>
</match>

Explanation of the Configuration

• Source (<source> block): Collects logs from /var/log/app.log and tags them as app.logs.
• Filters (<filter> blocks):
  • The first filter parses JSON fields from the message.
  • The second filter adds metadata like hostname and app_name.
  • The third filter uses a grep filter to include only logs with a severity of WARN or ERROR.
• Matches (<match> blocks):
  • The first match sends logs to Elasticsearch.
  • The second match writes logs to a local file in JSON format.
  • The third match sends critical logs to Slack for monitoring and alerts.

Using Tags for Routing

Tags (app.logs in the above example) are critical for routing logs through specific filters and matches. Ensure that your tag structure aligns with the filters and matches you want to apply.

Best Practices

1. Organize Filters and Matches: Keep filters and matches organized logically and document them to understand the log flow.
2. Use Buffering: Add buffering to matches, especially when sending data to external services like Elasticsearch or Slack, to handle errors and retries effectively.
3. Testing and Monitoring: Test each filter and match configuration separately to ensure they work as expected before deploying to production. Monitor Fluentd’s logs to troubleshoot issues quickly.

Got an article suggestion? Let us know

Explore more

Fluentd

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.