How to Get Filebeat to Ignore Certain Container Logs

To configure Filebeat to ignore certain container logs, you can use several methods depending on your needs. Here are some common approaches to achieve this:

1. Use exclude_files Option

If you want to ignore logs based on the file name or pattern, you can use the exclude_files option in the Filebeat configuration. This is useful for ignoring specific log files or patterns.

Example: Exclude Log Files Based on Pattern

filebeat.inputs:
  - type: log
    paths:
      - /var/lib/docker/containers/*/*.log
    exclude_files:
      - '*container_to_ignore.log'
      - '*another_pattern.log'

In this example:

• Logs from container_to_ignore.log and another_pattern.log are ignored.

2. Use ignore_older Option

You can use the ignore_older option to ignore logs older than a certain age. This can be helpful if you want to ignore logs from containers that are no longer active.

Example: Ignore Logs Older Than 24 Hours

filebeat.inputs:
  - type: log
    paths:
      - /var/lib/docker/containers/*/*.log
    ignore_older: 24h

3. Use processors for Filtering

Filebeat provides processors that can be used to drop or modify events based on conditions. The drop_event processor can be used to ignore logs based on specific criteria.

Example: Drop Logs Based on Field Value

If your logs include a specific field that identifies the container or log type you want to ignore, you can use a processor to drop those logs.

filebeat.inputs:
  - type: log
    paths:
      - /var/lib/docker/containers/*/*.log

processors:
  - drop_event:
      when:
        regexp:
          log.container.name: "container_to_ignore"

4. Use multiline Option for Multi-Line Logs

If you have multi-line logs and want to ignore certain patterns, you might need to configure the multiline option appropriately to ensure that unwanted logs are filtered out.

Example: Ignore Specific Multi-Line Log Patterns

filebeat.inputs:
  - type: log
    paths:
      - /var/lib/docker/containers/*/*.log
    multiline.pattern: '^\\d{4}-\\d{2}-\\d{2}T'
    multiline.negate: true
    multiline.match: after
    exclude_lines: ['^DEBUG']

In this example:

• Lines starting with DEBUG are excluded.

5. Use Logstash for Advanced Filtering

If Filebeat alone doesn’t provide the granularity you need, you can use Logstash for more complex filtering and routing.

Example: Drop Events Based on Field Values in Logstash

input {
  beats {
    port => 5044
  }
}

filter {
  if [container_name] == "container_to_ignore" {
    drop { }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "your-index-name-%{+YYYY.MM.dd}"
  }
}

6. Use Docker Labels for Filtering

If you use Docker and want to filter logs based on container labels, you might need to configure Filebeat or Logstash to use Docker labels.

Example: Docker Labels in Filebeat

Filebeat doesn’t directly filter by Docker labels, but you can use Logstash to handle this if you set up Docker logging with labels that you can filter on.

Summary

1. Use exclude_files to ignore logs based on file name patterns.
2. Use ignore_older to ignore logs older than a specified duration.
3. Use processors to drop events based on specific conditions or fields.
4. Use Logstash for more advanced filtering and processing if needed.
5. Consider Docker Labels if you need container-specific filtering.

These methods should help you configure Filebeat to ignore specific logs from containers based on your requirements.

Got an article suggestion? Let us know

Explore more

Filebeat

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.