Docker Apps Logging With Filebeat and Logstash

Logging Docker applications using Filebeat and Logstash is a robust solution for collecting, processing, and storing log data. This setup helps in centralizing logs and analyzing them effectively. Below is a step-by-step guide on how to configure Filebeat and Logstash for logging Docker applications.

Overview of the Setup

1. Filebeat: A lightweight shipper for forwarding and centralizing log data.
2. Logstash: A data processing pipeline that ingests data from multiple sources, transforms it, and sends it to a specified destination (like Elasticsearch).
3. Elasticsearch: The storage and search engine where logs are indexed and queried.

Step 1: Install and Configure Filebeat

1.1 Install Filebeat

If you haven't installed Filebeat yet, follow the instructions for your platform from the official Elastic documentation.

1.2 Configure Filebeat

Create or edit the Filebeat configuration file (e.g., filebeat.yml):

filebeat.inputs:
  - type: docker
    containers.ids:
      - '*'
    json.keys_under_root: true
    json.add_error_key: true

output.logstash:
  hosts: ["logstash:5044"]  # Specify your Logstash host and port

• Input Type: The docker input type allows Filebeat to read logs from Docker containers.
• JSON Configuration: The json options allow Filebeat to parse Docker's JSON log format directly.
• Output to Logstash: Specify the Logstash host and port where logs will be sent.

1.3 Run Filebeat

Run Filebeat using Docker or directly on the host:

filebeat -e -c /path/to/filebeat.yml

If using Docker, you can run Filebeat as follows:

docker run -d --name=filebeat \\
  --user=root \\
  --volume="/var/lib/docker/containers:/var/lib/docker/containers:ro" \\
  --volume="/var/run/docker.sock:/var/run/docker.sock:ro" \\
  --volume="/path/to/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro" \\
  docker.elastic.co/beats/filebeat:7.10.1 filebeat -e -strict.perms=false

Step 2: Install and Configure Logstash

2.1 Install Logstash

Follow the installation instructions for your platform from the official documentation.

2.2 Configure Logstash

Create or edit a Logstash configuration file (e.g., logstash.conf):

input {
  beats {
    port => 5044
  }
}

filter {
  # Optional filters to process logs
  if [stream] == "stdout" {
    mutate {
      add_field => { "log_type" => "stdout" }
    }
  } else if [stream] == "stderr" {
    mutate {
      add_field => { "log_type" => "stderr" }
    }
  }

  # Example: Parse JSON logs (if applicable)
  json {
    source => "message"
    target => "parsed_json"
    add_field => { "original_message" => "%{[message]}" }
  }
}

output {
  elasticsearch {
    hosts => ["<http://elasticsearch:9200>"]  # Specify your Elasticsearch host
    index => "docker-logs-%{+YYYY.MM.dd}"     # Index pattern for daily logs
  }
}

• Input Section: The beats input listens for logs from Filebeat.
• Filter Section: You can add filters to process the logs (e.g., mutating fields, parsing JSON).
• Output Section: The logs are sent to Elasticsearch with a specified index pattern.

Step 3: Run Logstash

Run Logstash with your configuration:

bin/logstash -f /path/to/logstash.conf

If using Docker, you can run Logstash as follows:

docker run -d --name=logstash \\
  -p 5044:5044 \\
  -v /path/to/logstash.conf:/usr/share/logstash/pipeline/logstash.conf \\
  docker.elastic.co/logstash/logstash:7.10.1

Step 4: Verify the Setup

1. Check Filebeat Logs: Ensure Filebeat is running and collecting logs.
2. Check Logstash Logs: Make sure Logstash is processing incoming logs.
3. Verify Elasticsearch: Query Elasticsearch to see if logs are being indexed.

curl -X GET "<http://localhost:9200/docker-logs-*/_search?pretty>"

Conclusion

By following these steps, you can successfully set up logging for Docker applications using Filebeat and Logstash. This configuration allows for centralized log collection and processing, making it easier to monitor and analyze logs across your Docker environment. Make sure to adjust the configurations to fit your specific needs and architecture.

Got an article suggestion? Let us know

Explore more

Logstash Filebeat

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.