How to Collect, Process, and Ship Log Data with Filebeat
Managing logs has become necessary in the ever-evolving landscape of modern computing. Operating systems, applications, and databases produce logs, which are invaluable for understanding system behavior, troubleshooting issues, and ensuring uninterrupted operations. To simplify the complex task of handling these logs, it becomes crucial to centralize the logs. This involves using a log shipper to collect and forward the logs to a central location.
Enter Filebeat, a powerful log shipper designed to streamline collecting, processing, and forwarding logs from diverse sources to various destinations. Developed with efficiency in mind, Filebeat ensures that managing logs is seamless and reliable. Its lightweight nature and ability to handle significant volumes of data make it a preferred choice among developers and system administrators.
In this comprehensive guide, you will explore the capabilities of Filebeat in depth using Docker containers. Starting with the basics, you'll set up Filebeat to collect logs from various sources. You'll then delve into the intricacies of processing these logs efficiently, gathering logs from Docker containers and forwarding them to different destinations for analysis and monitoring.
Prerequisites
Before you begin, ensure you have access to a system with a non-root user account with sudo
privileges. Additionally, ensure that you have Docker and Docker Compose installed on your system. If you're new to the concept of log shippers and their significance, take a moment to explore their advantages by reading this informative article.
With these prerequisites in place, create a dedicated project directory:
mkdir log-processing-stack
Navigate to the newly created directory:
cd log-processing-stack
Create a subdirectory for the demo application and move into the directory:
mkdir logify && cd logify
With these steps completed, you can create the demo logging application in the next section.
Developing a demo logging application
In this section, you'll build a basic logging application using the Bash scripting language. The application will generate logs at regular intervals, simulating a real-world scenario where applications produce log data.
In the logify
directory, create a logify.sh
file using your preferred text editor:
nano logify.sh
In your logify.sh
file, add the following code to generate logs:
#!/bin/bash
filepath="/var/log/logify/app.log"
create_log_entry() {
local info_messages=("Connected to database" "Task completed successfully" "Operation finished" "Initialized application")
local random_message=${info_messages[$RANDOM % ${#info_messages[@]}]}
local http_status_code=200
local ip_address="127.0.0.1"
local emailAddress="user@mail.com"
local level=30
local pid=$$
local ssn="407-01-2433"
local time=$(date +%s)
local log='{"status": '$http_status_code', "ip": "'$ip_address'", "level": '$level', "emailAddress": "'$emailAddress'", "msg": "'$random_message'", "pid": '$pid', "ssn": "'$ssn'", "timestamp": '$time'}'
echo "$log"
}
while true; do
log_record=$(create_log_entry)
echo "${log_record}" >> "${filepath}"
sleep 3
done
The create_log_entry()
function generates log records in JSON format, encompassing essential details like severity level, message, HTTP status code, and other crucial fields. In addition, it includes sensitive fields, such as email address, Social Security Number(SSN), and IP address, which have been deliberately included to demonstrate Filebeat ability to mask sensitive data in fields.
Next, the program enters an infinite loop that repeatedly invokes the create_log_entry()
function and writes the logs to a specified file in the /var/log/logify
directory.
After you finish adding the code, save the changes and make the script executable:
chmod +x logify.sh
Afterward, create the /var/log/logify
directory to store your application logs:
sudo mkdir -p /var/log/logify
Next, assign ownership of the /var/log/logify
directory to the currently logged-in user using the $USER
environment variable:
sudo chown -R $USER:$USER /var/log/logify/
Run the logify.sh
script in the background:
./logify.sh &
The &
symbol at the end of the command instructs the script to run in the background, allowing you to continue using the terminal for other tasks while the logging application runs independently.
When the program starts, it will display output that looks like this:
[1] 513579
Here, 513579
represents the process ID, which can be used to terminate the script later if needed.
To view the contents of the app.log
file, you can use the tail
command:
tail -n 4 /var/log/logify/app.log
This command displays the last 4 log entries in the app.log
file in JSON format:
{"status": 200, "ip": "127.0.0.1", "level": 30, "emailAddress": "user@mail.com", "msg": "Task completed successfully", "pid": 513579, "ssn": "407-01-2433", "timestamp": 1749023382}
{"status": 200, "ip": "127.0.0.1", "level": 30, "emailAddress": "user@mail.com", "msg": "Connected to database", "pid": 169516, "ssn": "407-01-2433", "timestamp": 1749023382}
{"status": 200, "ip": "127.0.0.1", "level": 30, "emailAddress": "user@mail.com", "msg": "Operation finished", "pid": 513579, "ssn": "407-01-2433", "timestamp": 1749023385}
{"status": 200, "ip": "127.0.0.1", "level": 30, "emailAddress": "user@mail.com", "msg": "Connected to database", "pid": 169516, "ssn": "407-01-2433", "timestamp": 1749023385}
You have now successfully created a logging application that generates sample log entries.
Setting up Filebeat with Docker
Instead of installing Filebeat directly on the system, we'll use the official Docker image. This approach provides better isolation, easier management, and consistent behavior across different environments.
First, Go back to log-processing-stack
directory and create a directory structure for Filebeat configurations:
cd ..
mkdir filebeat-config
The official Filebeat Docker image is available from the Elastic Docker registry at docker.elastic.co/beats/filebeat
, not from Docker Hub. You can find a list of all published Docker images and tags at www.docker.elastic.co.
To pull the latest Filebeat Docker image:
docker pull docker.elastic.co/beats/filebeat:9.0.1
For the absolute latest version, you can check the Elastic Docker registry for the most recent tag. As of this writing, version 9.0.1 is the latest.
You can verify the Filebeat version:
docker run --rm docker.elastic.co/beats/filebeat:9.0.1 filebeat version
If successful, the output will resemble:
filebeat version 9.0.1 (amd64), libbeat 9.0.1 [bce373f7dcd56a5575ad2c0ec40159722607e801 built 2025-04-29 23:44:32 +0000 UTC]
Now that Docker can run Filebeat, let's explore how it works.
How Filebeat works
Before you start using Filebeat, it's crucial to understand how it works. In this section, we'll explore its essential components and processes, ensuring you have a solid foundation before diving into practical usage:
Understanding how Filebeat works mainly involves familiarizing yourself with the following components:
Harvesters: harvesters are responsible for reading the contents of a file line by line. A harvester is initiated for each file when Filebeat is configured to monitor specific log files. These harvesters not only read the log data but also manage opening and closing files. By reading files incrementally, line by line, harvesters ensure that newly appended log data is efficiently collected and forwarded for processing.
Inputs: inputs serve as the bridge between harvesters and the data sources. They are responsible for managing the harvesters and locating all the sources from which Filebeat needs to read log data. Inputs can be configured for various sources, such as log files, containers, or system logs. Users can specify which files or locations Filebeat should monitor by defining inputs.
After Filebeat reads the log data, the log events are transformed or enriched with data. And then finally sent to the specified destinations.
To put into practice, you can specify this behavior in a configuration file:
filebeat.inputs:
. . .
processors:
. . .
output.plugin_name:
. . .
Let's now look into each section in detail:
filebeat.inputs
: the input sources that a Filebeat instance should monitor.processors
: enrich, modify, or filter data before it's sent to the output.output.plugin_name
: the output destination where Filebeat should forward the log data.
Each of these directives requires you to specify a plugin that carries out its respective task.
Now, let's explore some inputs, processors, and outputs that can be used with Filebeat.
Filebeat input plugins
Filebeat provides a range of inputs plugins, each tailored to collect log data from specific sources:
- container: collect container logs.
- filestream: actively reads lines from log files.
- syslog: fetches log entries from Syslog.
- httpjson: read log messages from a RESTful API.
Filebeat output plugins
Filebeat provides a variety of outputs plugins, enabling you to send your collected log data to diverse destinations:
- File: writes log events to files.
- Elasticsearch: enables Filebeat to forward logs to Elasticsearch using its HTTP API.
- Kafka: delivers log records to Apache Kafka.
- Logstash: sends logs directly to Logstash.
Filebeat modules plugins
Filebeat streamlines log processing through its modules, providing pre-configured setups designed for specific log formats. These modules enable you to effortlessly ingest, parse, and enrich log data without requiring extensive manual configuration. Here are a few available modules that can significantly simplify your log processing workflow:
Getting started with Filebeat using Docker
Now that you understand the workings of Filebeat, let's configure it to read log entries from a file and display them on the console using Docker.
First, create the Filebeat configuration file in the filebeat-config
directory:
cd filebeat-config
nano filebeat.yml
Add the following configuration:
filebeat.inputs:
- type: filestream
id: logify-logs
paths:
- /var/log/logify/app.log
output.console:
pretty: true
In the filebeat.inputs
section, you specify that Filebeat should read logs from a file using the filestream
plugin. The paths
parameter indicates the path to the log file that Filebeat will monitor, set here as /var/log/logify/app.log
. The id
field is required for filestream inputs and must be unique.
The output.console
section sends the collected log data to the console. The pretty: true
parameter ensures that log entries are presented in a readable and well-structured format when shown on the console.
Once you've added these configurations, save the file.
Before executing Filebeat, you need to set the correct file permissions. Filebeat requires that configuration files are only writable by the owner:
chmod 600 filebeat.yml
Before executing Filebeat, it's essential to verify the configuration file syntax to identify and rectify any errors using Docker:
docker run --rm \
-v $(pwd)/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro \
docker.elastic.co/beats/filebeat:9.0.1 \
filebeat test config
If the configuration file is correct, you should see the following output:
Config OK
Now, run Filebeat using Docker with the standalone approach:
docker run --rm \
-v $(pwd)/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro \
-v /var/log/logify:/var/log/logify:ro \
--name filebeat-basic \
docker.elastic.co/beats/filebeat:9.0.1
The command above:
- --rm
: Removes the container when it stops
- -v $(pwd)/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
: Mounts the configuration file as read-only
- -v /var/log/logify:/var/log/logify:ro
: Mounts the log directory as read-only
- --name filebeat-basic
: Names the container for easy reference
As Filebeat starts running, it will display log entries similar to the following:
...
{
"@timestamp": "2025-06-04T07:52:54.769Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "9.0.1"
},
"log": {
"offset": 779629,
"file": {
"path": "/var/log/logify/app.log",
"device_id": "2049",
"inode": "259470",
"fingerprint": "337d323bb6858c439b81fec30abaccc82f8576477737a881801695d2ad617b1f"
}
},
"message": "{\"status\": 200, \"ip\": \"127.0.0.1\", \"level\": 30, \"emailAddress\": \"user@mail.com\", \"msg\": \"Task completed successfully\", \"pid\": 169516, \"ssn\": \"407-01-2433\", \"timestamp\": 1748878010}",
"input": {
"type": "filestream"
},
"ecs": {
"version": "8.0.0"
},
"host": {
"name": "645301c4e5a2"
},
"agent": {
"id": "c9c201f5-b93b-4719-976e-23eebfbbc91e",
"name": "645301c4e5a2",
"type": "filebeat",
"version": "9.0.1",
"ephemeral_id": "714a7298-45e8-4f57-bc8e-6426bf1a7d7d"
}
}
...
Filebeat now displays log messages in the console. The log events from the Bash script are under the message
field, and Filebeat has added additional fields to provide context. You can now stop Filebeat by pressing CTRL + C
.
Having successfully configured Filebeat to read and forward logs to the console, the next section will focus on data transformation.
Transforming logs with Filebeat
When Filebeat collects data, you can process it before sending it to the output. You can enrich it with new fields, parse the data, and remove or redact sensitive fields to ensure data privacy.
In this section, you'll transform the logs in the following ways:
- Parsing JSON logs.
- Removing unwanted fields.
- Adding new fields.
- Masking sensitive data.
Parsing JSON logs with Filebeat
As the demo logging application generates logs in JSON format, it's essential to parse them correctly for structured analysis.
Let's examine an example log event from the previous section:
{
...
"message": "{\"status\": 200, \"ip\": \"127.0.0.1\", \"level\": 30, \"emailAddress\": \"user@mail.com\", \"msg\": \"Task completed successfully\", \"pid\": 169516, \"ssn\": \"407-01-2433\", \"timestamp\": 1748878010}",
"input": {
"type": "filestream"
},
...
}
In the message
field, double quotes surround the log event, and many fields are escaped with backslashes. This is not valid JSON; the message
field contents have been converted to a string.
To parse the log event as valid JSON, update the configuration file:
nano filebeat.yml
Update the configuration to add JSON parsing:
filebeat.inputs:
- type: filestream
id: logify-logs
paths:
- /var/log/logify/app.log
processors:
- decode_json_fields:
fields: ["message"]
target: ""
output.console:
pretty: true
In the snippet above, you configure a decode_json_fields
processor to decode JSON-encoded data in each log entry's message
field and attach it to the log event.
Run Filebeat with the updated configuration:
docker run --rm \
-v $(pwd)/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro \
-v /var/log/logify:/var/log/logify:ro \
--name filebeat-json \
docker.elastic.co/beats/filebeat:9.0.1
{
"@timestamp": "2025-06-04T07:55:03.658Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "9.0.1"
},
"pid": 169516,
"ssn": "407-01-2433",
"log": {
"offset": 699725,
"file": {
"path": "/var/log/logify/app.log",
"device_id": "2049",
"inode": "259470",
"fingerprint": "337d323bb6858c439b81fec30abaccc82f8576477737a881801695d2ad617b1f"
}
},
"message": "{\"status\": 200, \"ip\": \"127.0.0.1\", \"level\": 30, \"emailAddress\": \"user@mail.com\", \"msg\": \"Task completed successfully\", \"pid\": 169516, \"ssn\": \"407-01-2433\", \"timestamp\": 1748876646}",
"input": {
"type": "filestream"
},
"ecs": {
"version": "8.0.0"
},
"host": {
"name": "deffe2f97ae8"
},
"agent": {
"name": "deffe2f97ae8",
"type": "filebeat",
"version": "9.0.1",
"ephemeral_id": "354e0fef-db29-48bf-b9fc-96a3bf5d011b",
"id": "28d7a11c-5619-44b6-a094-6b59bb9c2313"
},
"timestamp": 1748876646,
"ip": "127.0.0.1",
"msg": "Task completed successfully",
"status": 200,
"level": 30,
"emailAddress": "user@mail.com"
}
...
In the output, you will see that all properties in the message
field, such as msg
, ip
, etc., have been added to the log event.
Now that you can parse JSON logs, you will modify attributes on a log event.
Adding and removing fields with Filebeat
The log event contains a sensitive emailAddress
field that needs to be protected. In this section, you'll remove the emailAddress
field and add a new field to the log event to provide more context.
Update the configuration file:
nano filebeat.yml
Update the configuration to add field manipulation:
filebeat.inputs:
- type: filestream
id: logify-logs
paths:
- /var/log/logify/app.log
processors:
- decode_json_fields:
fields: ["message"]
target: ""
- drop_fields:
fields: ["emailAddress", "message"]
- add_fields:
fields:
env: "development" # Add a new 'env' field set to "development"
output.console:
pretty: true
To modify the log event, you add the drop_fields
processor, which has a field
option that takes a list of fields to be removed, including the sensitive emailAddress
field and the message
field. You remove the message
field because after parsing the data, properties from the message
field were incorporated into the log event, rendering the original message
field obsolete.
After writing the code, run the configuration:
docker run --rm \
-v $(pwd)/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro \
-v /var/log/logify:/var/log/logify:ro \
--name filebeat-transform \
docker.elastic.co/beats/filebeat:9.0.1
Upon running Filebeat, you will notice that the emailAddress
field has been successfully removed, and a new env
field has been added to the log event:
d
{
....
"agent": {
"version": "9.0.1",
"ephemeral_id": "875d13fe-aa4e-491f-84c7-94ac2ff3378d",
"id": "525d8aa6-26f4-46cf-9941-ed9dc7c849e6",
"name": "1f1af5b9fc53",
"type": "filebeat"
},
"ssn": "407-01-2433",
"level": 30,
"pid": 169516,
"log": {
"offset": 658520,
"file": {
"path": "/var/log/logify/app.log",
"device_id": "2049",
"inode": "259470",
"fingerprint": "337d323bb6858c439b81fec30abaccc82f8576477737a881801695d2ad617b1f"
}
},
"input": {
"type": "filestream"
},
"timestamp": 1748875942,
"status": 200,
"ip": "127.0.0.1",
"fields": {
"env": "development"
},
"msg": "Operation finished"
}
...
Now that you can enrich and remove unwanted fields, you will work with conditional statements next.
Working with conditional statements in Filebeat
Filebeat allows you to check a condition and add a field when it evaluates to true. In this section, you will check if the status
value equals 200
, and if the condition is met, you will add an is_successful
field to the log event.
Update the configuration file:
nano filebeat.yml
Add conditional field processing:
filebeat.inputs:
- type: filestream
id: logify-logs
paths:
- /var/log/logify/app.log
processors:
- decode_json_fields:
fields: ["message"]
target: ""
- drop_fields:
fields: ["emailAddress", "message"] # Remove the 'emailAddress' field
- add_fields:
fields:
env: "development" # Add a new 'env' field set to "development"
- add_fields:
when:
equals:
status: 200
target: ""
fields:
is_successful: true
output.console:
pretty: true
The when
option checks if the status
field value equals 200
. If true, the is_successful
field is added to the log event.
Now, run the configuration:
docker run --rm \
-v $(pwd)/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro \
-v /var/log/logify:/var/log/logify:ro \
--name filebeat-conditional \
docker.elastic.co/beats/filebeat:9.0.1
Filebeat will yield output that looks closely to this:
{
...
"host": {
"name": "a3bc504aa467"
},
"ssn": "407-01-2433",
"is_successful": true,
"input": {
"type": "filestream"
},
"status": 200,
"ip": "127.0.0.1"
}
...
In the output, the is_successful
field has been added to the log entries with an HTTP status code of 200.
That takes care of adding a new field based on a condition.
Redacting sensitive data with Filebeat
Earlier in the article, you removed the emailAddress
field to ensure data privacy. However, sensitive fields such as IP addresses and Social Security Numbers (SSN) remain in the log event. Moreover, sensitive data can be inadvertently added to log events by other developers within an organization. Redacting data that match specific patterns allows you to mask any sensitive information without needing to remove entire fields, ensuring the message's significance is preserved.
Update the configuration file to add sensitive data redaction:
nano filebeat.yml
Add the following configuration:
filebeat.inputs:
- type: filestream
id: logify-logs
paths:
- /var/log/logify/app.log
processors:
- script:
lang: javascript
id: redact-sensitive-info
source: |
function process(event) {
// Redact SSNs (e.g., 123-45-6789) from the "message" field
event.Put("message", event.Get("message").replace(/\d{3}-\d{2}-\d{4}/g, "[REDACTED-SSN]"));
// Redact IP addresses (e.g., 192.168.1.1) from the "message" field
event.Put("message", event.Get("message").replace(/\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b/g, "[REDACTED-IP]"));
}
- decode_json_fields:
fields: ["message"]
target: ""
- drop_fields:
fields: ["emailAddress"] # Remove the 'emailAddress' field
- add_fields:
fields:
env: "development" # Add a new 'env' field set to "development"
- add_fields:
when:
equals:
status: 200
target: ""
fields:
is_successful: true
output.console:
pretty: true
In the added code, you define a script written in JavaScript that redacts sensitive information from a log event. The script uses regular expressions to identify SSNs and IP addresses, replacing them with [REDACTED-SSN]
and [REDACTED-IP]
, respectively.
Next, run the configuration:
docker run --rm \
-v $(pwd)/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro \
-v /var/log/logify:/var/log/logify:ro \
--name filebeat-redact \
docker.elastic.co/beats/filebeat:9.0.1
...
{
"@timestamp": "2025-06-04T08:13:18.496Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "9.0.1"
},
"fields": {
"env": "development"
},
"input": {
"type": "filestream"
},
"ip": "[REDACTED-IP]",
"status": 200,
"msg": "Operation finished",
"pid": 169516,
"timestamp": 1748871122,
"log": {
"file": {
"path": "/var/log/logify/app.log",
"device_id": "2049",
"inode": "259470",
"fingerprint": "337d323bb6858c439b81fec30abaccc82f8576477737a881801695d2ad617b1f"
},
"offset": 376373
},
"ecs": {
"version": "8.0.0"
},
"host": {
"name": "6079c3618796"
},
"agent": {
"type": "filebeat",
"version": "9.0.1",
"ephemeral_id": "4e8cea1f-bd78-4068-b016-1b8e36e768af",
"id": "a4a5b3a4-ac08-4488-aac0-a0251c0d4ec7",
"name": "6079c3618796"
},
"level": 30,
"is_successful": true,
"ssn": "[REDACTED-SSN]"
}
...
The log events in the output will now have the IP address and SSN fields redacted.
In scenarios where you have a field like the following:
{..., "privateInfo": "This is a sample message with SSN: 123-45-6789 and IP: 192.168.0.1"}
After processing with Filebeat, only the sensitive portions will be removed, and the log event will appear as follows:
{..., "privateInfo": "This is a sample message with SSN: [REDACTED-SSN] and IP: [REDACTED-IP]"}
The sensitive portions have now been redacted, and the context of the log message remains intact.
You can now stop the running Docker container by pressing CTRL + C
and stop the logify.sh
program.
To stop the bash script, obtain the process ID:
jobs -l | grep "logify"
[1]+ 513579 Running ./logify.sh & (wd: ~/log-processing-stack/logify)
Substitute the process ID in the kill
command:
kill -9 513579
The program will now be terminated.
Having successfully redacted sensitive fields, you can now collect logs from Docker containers using Filebeat and centralize them for further analysis and monitoring.
Collecting logs from Docker containers and centralizing logs
In this section, you will containerize the Bash script and use the Nginx hello world Docker image, preconfigured to generate JSON Nginx logs for each incoming request. Subsequently, you will create a Filebeat container to gather logs from both containers and centralize them to Better Stack for analysis.
Dockerizing the Bash script
In this section, you'll containerize the Bash script that generates log data. This step allows you to encapsulate the script and its dependencies.
Navigate back to the logify directory and create a Dockerfile
:
cd ../logify
nano Dockerfile
In your Dockerfile
, add the following code:
FROM ubuntu:latest
COPY . .
RUN chmod +x logify.sh
RUN mkdir -p /var/log/logify
RUN ln -sf /dev/stdout /var/log/logify/app.log
CMD ["./logify.sh"]
In the first line, the latest version of Ubuntu is specified as the base image. Next, the script is copied into the container, made executable, and a directory to store log files is created. Subsequently, any data written to /var/log/logify/app.log
is redirected to the standard output so that it can be viewed using the docker logs
command. Finally, you specify the command to run when the container is first launched.
Save and exit the file after making these changes. Then, change into the parent project directory:
cd ..
Next, create a docker-compose.yml
file to define the services and volumes for your Docker containers:
nano docker-compose.yml
Define the Bash script and Nginx services:
services:
logify-script:
build:
context: ./logify
image: logify:latest
container_name: logify
nginx:
image: betterstackcommunity/nginx-helloworld:latest
logging:
driver: json-file
container_name: nginx
ports:
- '80:80'
In this Docker Compose file, you define two services: logify-script
and nginx
. The logify-script
service is built from the ./logify
directory context, creating an image tagged as logify:latest
. The nginx
service uses the latest version of the nginx-helloworld image and the json-file
logging driver for logging purposes. Additionally, port 80
on the host is mapped to port 80
within the container. Ensure no other services use port 80
to prevent conflicts.
To build the logify-script
service image and start the containers for each defined service, use the following command:
docker compose up -d
The -d
option puts the services in the background.
Now, check if the services are running:
docker compose ps
You should see a "running" status under the "STATUS" column for both containers, which should look like this:
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
logify logify:latest "./logify.sh" logify-script 26 seconds ago Up 26 seconds
nginx betterstackcommunity/nginx-helloworld:latest "/runner.sh nginx" nginx 26 seconds ago Up 26 seconds 0.0.0.0:80->80/tcp, [::]:80->80/tcp
With the containers running, use the curl
command to send HTTP requests to the Nginx service:
curl http://localhost:80/?[1-5]
View all the logs from both containers with the following command:
docker compose logs
nginx | {"timestamp":"2025-06-04T08:17:18+00:00","pid":"8","remote_addr":"172.18.0.1","remote_user":"","request":"GET /?1 HTTP/1.1","status": "200","body_bytes_sent":"11109","request_time":"0.000","http_referrer":"","http_user_agent":"curl/8.5.0","time_taken_ms":"1749025038.755"}
...
logify | {"status": 200, "ip": "127.0.0.1", "level": 30, "emailAddress": "user@mail.com", "msg": "Connected to database", "pid": 1, "ssn": "407-01-2433", "timestamp": 1749025042}
The output displays logs produced from both containers.
Now that the Bash script and Nginx services are running and generating logs, you can collect and centralize these logs using Filebeat.
Setting up Better Stack source
Before configuring Filebeat to forward logs, let's set up Better Stack to receive them.
Start by creating a free Better Stack account. Once you've logged in, navigate to the Sources section and click the Connect source button:
Give the source a meaningful name like "Logify App Logs", select Docker as the platform, scroll to the bottom, and click Connect Source:
After creating the source, you'll receive a source token (e.g., qU73jvQjZrNFHimZo4miLdxF
) and an ingestion host (e.g., s1315908.eu-nbg-2.betterstackdata.com
).
Save both—these are required when setting up log forwarding in your Filebeat config:
Keep the token and injection host safe as you'll need them for the Filebeat configuration.
Now you'll configure Filebeat to collect logs from the logify container. Update the filebeat.yml
configuration file to collect logs from Docker containers:
cd filebeat-config
nano filebeat.yml
Replace the existing configuration with the following, replacing <your_source_token>
with your actual source token and <your_ingestion_host>
with your ingestion host:
filebeat.autodiscover:
providers:
- type: docker
labels.dedot: true
templates:
- condition:
contains:
docker.container.image: "logify"
config:
- type: filestream
id: container-${data.docker.container.id}
prospector.scanner.symlinks: true
parsers:
- container: ~
paths:
- /var/lib/docker/containers/${data.docker.container.id}/*.log
processors:
- decode_json_fields:
fields: ["message"]
target: ""
output.elasticsearch:
hosts: 'https://<your_ingestion_host>:443'
path: '/elastic'
headers:
X-Better-Stack-Source-Token: '<your_source_token>'
In this configuration, you set up Filebeat's automatic log discovery to collect logs from Docker containers whose image names contain the substring logify
. This corresponds to the container defined under the logify-script
service. Filebeat uses the container
input to read Docker logs specified under paths. Additionally, a processor is added to decode JSON fields.
Now, add the Filebeat service to your docker-compose.yml
file:
cd ..
nano docker-compose.yml
Update the file to include the Filebeat service:
services:
logify-script:
build:
context: ./logify
image: logify:latest
container_name: logify
nginx:
image: betterstackcommunity/nginx-helloworld:latest
logging:
driver: json-file
container_name: nginx
ports:
- '80:80'
filebeat-logify:
image: docker.elastic.co/beats/filebeat:9.0.1
container_name: filebeat-logify
user: root
command:
- "-e"
- "--strict.perms=false"
volumes:
- ./filebeat-config/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
- /var/lib/docker/containers:/var/lib/docker/containers:ro
- /var/run/docker.sock:/var/run/docker.sockock
In this updated configuration, the filebeat-logify
service uses the official Filebeat Docker image, with the user set to root
. The Filebeat configuration is mounted from the filebeat-config
directory.
Start the Filebeat service:
docker compose up -d
Check that all services are running:
docker compose ps
Wait a bit, then revisit Better Stack to verify that Filebeat is forwarding logs. The source should now read “Logs received!”. After that, click Live tail to view your logs in real time:
Navigate to your "Logify logs" source and you should see logs appearing in the live tail:
The logs should show the processed JSON data from your logify application, confirming that Filebeat is successfully collecting and forwarding the logs.
Adding Nginx logs
Now that you have successfully forwarded the Bash script logs, it's time to add Nginx service logs as well. First, create an additional source named "Nginx logs" in Better Stack by following the same steps you did previously. Make sure to copy both the source token and ingestion host to a safe place.
Now you'll need to create a separate Filebeat configuration and service to handle Nginx logs. Create a new configuration file:
cd filebeat-config
nano filebeat-nginx.yml
Add the following configuration, replacing <your_nginx_source_token>
with your actual nginx source token and <your_nginx_ingestion_host>
with your nginx ingestion host:
filebeat.autodiscover:
providers:
- type: docker
labels.dedot: true
templates:
- condition:
contains:
docker.container.image: "betterstackcommunity/nginx-helloworld"
config:
- type: filestream
id: container-${data.docker.container.id}
prospector.scanner.symlinks: true
parsers:
- container: ~
paths:
- /var/lib/docker/containers/${data.docker.container.id}/*.log
processors:
- decode_json_fields:
fields: ["message"]
target: ""
output.elasticsearch:
hosts: 'https://<your_nginx_ingestion_host>:443'
path: '/elastic'
headers:
X-Better-Stack-Source-Token: '<your_nginx_source_token>'
Now add the nginx Filebeat service to your docker-compose.yml
:
cd ..
nano docker-compose.yml
Add the nginx Filebeat service:
services:
logify-script:
build:
context: ./logify
image: logify:latest
container_name: logify
nginx:
image: betterstackcommunity/nginx-helloworld:latest
logging:
driver: json-file
container_name: nginx
ports:
- '80:80'
filebeat-logify:
image: docker.elastic.co/beats/filebeat:9.0.1
container_name: filebeat-logify
user: root
command:
- "-e"
- "--strict.perms=false"
volumes:
- ./filebeat-config/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
- /var/lib/docker/containers:/var/lib/docker/containers:ro
- /var/run/docker.sock:/var/run/docker.sock
filebeat-nginx:
image: docker.elastic.co/beats/filebeat:9.0.1
container_name: filebeat-nginx
user: root
command:
- "-e"
- "--strict.perms=false"
volumes:
- ./filebeat-config/filebeat-nginx.yml:/usr/share/filebeat/filebeat.yml:ro
- /var/lib/docker/containers:/var/lib/docker/containers:ro
- /var/run/docker.sock:/var/run/docker.sock
Start the services:
docker compose up -d
Send a few requests to the Nginx service to generate logs:
curl http://localhost:80/?[1-5]
Now go back to Better Stack to confirm that the "Nginx logs" source is receiving the logs. Navigate to your "Nginx logs" source and your source will update to show "Logs received!". Click Live tail to view your logs in real-time:
Click on any log entry to expand it and view more detailed information:
Both sources should now be receiving their respective logs, demonstrating successful log centralization with separate sources for different container types.
Monitoring Filebeat health with Better Stack
Filebeat doesn't have a built-in /health
endpoint for external monitoring of its instance's health. However, you can configure an endpoint for metrics. Doing so allows you to externally monitor Filebeat to determine whether it's up or down. In this tutorial, you will enable the HTTP endpoint for the filebeat-logify
service.
Open the filebeat.yml
configuration file:
cd filebeat-config
nano filebeat.yml
Add the following lines at the top of the file:
http.enabled: true
http.host: 0.0.0.0
http.port: 5066
filebeat.autodiscover:
providers:
...
output.elasticsearch:
hosts: 'https://s1332781.eu-nbg-2.betterstackdata.com:443'
path: '/elastic'
headers:
X-Better-Stack-Source-Token: 'T3jWxnJJowffDjDkriV5siJK'
The metric endpoint will be enabled and exposed on port 5066
.
Next, update the Docker Compose file to map port 5066
between the host and the container:
cd ..
nano docker-compose.yml
Update the filebeat-logify
service to expose the metrics port:
filebeat-logify:
image: docker.elastic.co/beats/filebeat:9.0.1
container_name: filebeat-logify
user: root
ports:
- "5066:5066"
command:
- "-e"
- "--strict.perms=false"
volumes:
- ./filebeat-config/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
- /var/lib/docker/containers:/var/lib/docker/containers:ro
- /var/run/docker.sock:/var/run/docker.sock
Restart the Filebeat service to apply the changes:
docker compose up -d
Verify that the Filebeat metrics endpoint is functioning properly:
curl -XGET 'localhost:5066/?pretty'
{
"beat": "filebeat",
"binary_arch": "amd64",
"build_commit": "bce373f7dcd56a5575ad2c0ec40159722607e801",
"build_time": "2025-04-29T23:44:32.000Z",
"elastic_licensed": true,
"ephemeral_id": "5828abd5-8591-48d5-9d3a-15b6281cd504",
"gid": "0",
"hostname": "6fc86bf74515",
"name": "6fc86bf74515",
"uid": "0",
"username": "root",
"uuid": "61654154-4353-478e-88ac-3556aa7f7dd5",
"version": "9.0.1"
}
Next, sign into Better Stack uptime monitoring.
On the Monitors page, click the Create monitor button:
Choose the preferred method to trigger Better Stack, provide your server's IP address or domain name on port 5066
, and click the Create monitor button:
Better Stack will start monitoring the Filebeat endpoint and provide performance insights:
Now, let's observe how Better Stack responds when Filebeat stops working by stopping all the services:
docker compose stop
After some time has passed, return to Better Stack to observe that the status has been updated to "Down":
If you configured Better Stack to send you an email, check your email inbox. You will receive an email alert:
That takes care of monitoring Filebeat using Better Stack.
Final thoughts
In this tutorial, you learned how to use Filebeat with Docker to integrate with Docker containers, Nginx, and Better Stack to manage logs. You started by reading logs from a file and displaying them in a console using Docker containers. Then, you explored various ways to transform log messages using Docker-based Filebeat configurations. After that, you collected logs from multiple Docker containers and forwarded them to Better Stack. Finally, you monitored Filebeat's health using Better Stack and received alerts in case of issues.
As a next step, refer to the Filebeat documentation for more in-depth information. To learn more about Docker and Docker Compose, explore their respective documentation pages: Docker and Docker Compose. To enhance your Docker logging knowledge, check out our comprehensive guide.
Apart from Filebeat, there are other log shippers available that you can explore. Check out the log shippers guide to learn about them.
Thanks for reading, and happy logging!
Make your mark
Join the writer's program
Are you a developer and love writing and sharing your knowledge with the world? Join our guest writing program and get paid for writing amazing technical guides. We'll get them to the right readers that will appreciate them.
Write for us
Build on top of Better Stack
Write a script, app or project on top of Better Stack and share it with the world. Make a public repository and share it with us at our email.
community@betterstack.comor submit a pull request and help us build better products for everyone.
See the full list of amazing projects on github