How to Collect, Process, and Ship Log Data with Filebeat
Managing logs has become necessary in the ever-evolving landscape of modern computing. Operating systems, applications, and databases produce logs, which are invaluable for understanding system behavior, troubleshooting issues, and ensuring uninterrupted operations. To simplify the complex task of handling these logs, it becomes crucial to centralize the logs. This involves using a log shipper to collect and forward the logs to a central location.
Enter Filebeat, a powerful log shipper designed to streamline collecting, processing, and forwarding logs from diverse sources to various destinations. Developed with efficiency in mind, Filebeat ensures that managing logs is seamless and reliable. Its lightweight nature and ability to handle significant volumes of data make it a preferred choice among developers and system administrators.
In this comprehensive guide, you will explore the capabilities of Filebeat in depth using Docker containers. Starting with the basics, you'll set up Filebeat to collect logs from various sources. You'll then delve into the intricacies of processing these logs efficiently, gathering logs from Docker containers and forwarding them to different destinations for analysis and monitoring.
Prerequisites
Before you begin, ensure you have access to a system with a non-root user account with sudo privileges. Additionally, ensure that you have Docker and Docker Compose installed on your system. If you're new to the concept of log shippers and their significance, take a moment to explore their advantages by reading this informative article.
With these prerequisites in place, create a dedicated project directory:
Navigate to the newly created directory:
Create a subdirectory for the demo application and move into the directory:
With these steps completed, you can create the demo logging application in the next section.
Developing a demo logging application
In this section, you'll build a basic logging application using the Bash scripting language. The application will generate logs at regular intervals, simulating a real-world scenario where applications produce log data.
In the logify directory, create a logify.sh file using your preferred text editor:
In your logify.sh file, add the following code to generate logs:
The create_log_entry() function generates log records in JSON format, encompassing essential details like severity level, message, HTTP status code, and other crucial fields. In addition, it includes sensitive fields, such as email address, Social Security Number(SSN), and IP address, which have been deliberately included to demonstrate Filebeat ability to mask sensitive data in fields.
Next, the program enters an infinite loop that repeatedly invokes the create_log_entry() function and writes the logs to a specified file in the /var/log/logify directory.
After you finish adding the code, save the changes and make the script executable:
Afterward, create the /var/log/logify directory to store your application logs:
Next, assign ownership of the /var/log/logify directory to the currently logged-in user using the $USER environment variable:
Run the logify.sh script in the background:
The & symbol at the end of the command instructs the script to run in the background, allowing you to continue using the terminal for other tasks while the logging application runs independently.
When the program starts, it will display output that looks like this:
Here, 513579 represents the process ID, which can be used to terminate the script later if needed.
To view the contents of the app.log file, you can use the tail command:
This command displays the last 4 log entries in the app.log file in JSON format:
You have now successfully created a logging application that generates sample log entries.
Setting up Filebeat with Docker
Instead of installing Filebeat directly on the system, we'll use the official Docker image. This approach provides better isolation, easier management, and consistent behavior across different environments.
First, Go back to log-processing-stack directory and create a directory structure for Filebeat configurations:
The official Filebeat Docker image is available from the Elastic Docker registry at docker.elastic.co/beats/filebeat, not from Docker Hub. You can find a list of all published Docker images and tags at www.docker.elastic.co.
To pull the latest Filebeat Docker image:
For the absolute latest version, you can check the Elastic Docker registry for the most recent tag. As of this writing, version 9.0.1 is the latest.
You can verify the Filebeat version:
If successful, the output will resemble:
Now that Docker can run Filebeat, let's explore how it works.
How Filebeat works
Before you start using Filebeat, it's crucial to understand how it works. In this section, we'll explore its essential components and processes, ensuring you have a solid foundation before diving into practical usage:
Understanding how Filebeat works mainly involves familiarizing yourself with the following components:
Harvesters: harvesters are responsible for reading the contents of a file line by line. A harvester is initiated for each file when Filebeat is configured to monitor specific log files. These harvesters not only read the log data but also manage opening and closing files. By reading files incrementally, line by line, harvesters ensure that newly appended log data is efficiently collected and forwarded for processing.
Inputs: inputs serve as the bridge between harvesters and the data sources. They are responsible for managing the harvesters and locating all the sources from which Filebeat needs to read log data. Inputs can be configured for various sources, such as log files, containers, or system logs. Users can specify which files or locations Filebeat should monitor by defining inputs.
After Filebeat reads the log data, the log events are transformed or enriched with data. And then finally sent to the specified destinations.
To put into practice, you can specify this behavior in a configuration file:
Let's now look into each section in detail:
filebeat.inputs: the input sources that a Filebeat instance should monitor.processors: enrich, modify, or filter data before it's sent to the output.output.plugin_name: the output destination where Filebeat should forward the log data.
Each of these directives requires you to specify a plugin that carries out its respective task.
Now, let's explore some inputs, processors, and outputs that can be used with Filebeat.
Filebeat input plugins
Filebeat provides a range of inputs plugins, each tailored to collect log data from specific sources:
- container: collect container logs.
- filestream: actively reads lines from log files.
- syslog: fetches log entries from Syslog.
- httpjson: read log messages from a RESTful API.
Filebeat output plugins
Filebeat provides a variety of outputs plugins, enabling you to send your collected log data to diverse destinations:
- File: writes log events to files.
- Elasticsearch: enables Filebeat to forward logs to Elasticsearch using its HTTP API.
- Kafka: delivers log records to Apache Kafka.
- Logstash: sends logs directly to Logstash.
Filebeat modules plugins
Filebeat streamlines log processing through its modules, providing pre-configured setups designed for specific log formats. These modules enable you to effortlessly ingest, parse, and enrich log data without requiring extensive manual configuration. Here are a few available modules that can significantly simplify your log processing workflow:
Getting started with Filebeat using Docker
Now that you understand the workings of Filebeat, let's configure it to read log entries from a file and display them on the console using Docker.
First, create the Filebeat configuration file in the filebeat-config directory:
Add the following configuration:
In the filebeat.inputs section, you specify that Filebeat should read logs from a file using the filestream plugin. The paths parameter indicates the path to the log file that Filebeat will monitor, set here as /var/log/logify/app.log. The id field is required for filestream inputs and must be unique.
The output.console section sends the collected log data to the console. The pretty: true parameter ensures that log entries are presented in a readable and well-structured format when shown on the console.
Once you've added these configurations, save the file.
Before executing Filebeat, you need to set the correct file permissions. Filebeat requires that configuration files are only writable by the owner:
Before executing Filebeat, it's essential to verify the configuration file syntax to identify and rectify any errors using Docker:
If the configuration file is correct, you should see the following output:
Now, run Filebeat using Docker with the standalone approach:
The command above:
- --rm: Removes the container when it stops
- -v $(pwd)/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro: Mounts the configuration file as read-only
- -v /var/log/logify:/var/log/logify:ro: Mounts the log directory as read-only
- --name filebeat-basic: Names the container for easy reference
As Filebeat starts running, it will display log entries similar to the following:
Filebeat now displays log messages in the console. The log events from the Bash script are under the message field, and Filebeat has added additional fields to provide context. You can now stop Filebeat by pressing CTRL + C.
Having successfully configured Filebeat to read and forward logs to the console, the next section will focus on data transformation.
Transforming logs with Filebeat
When Filebeat collects data, you can process it before sending it to the output. You can enrich it with new fields, parse the data, and remove or redact sensitive fields to ensure data privacy.
In this section, you'll transform the logs in the following ways:
- Parsing JSON logs.
- Removing unwanted fields.
- Adding new fields.
- Masking sensitive data.
Parsing JSON logs with Filebeat
As the demo logging application generates logs in JSON format, it's essential to parse them correctly for structured analysis.
Let's examine an example log event from the previous section:
In the message field, double quotes surround the log event, and many fields are escaped with backslashes. This is not valid JSON; the message field contents have been converted to a string.
To parse the log event as valid JSON, update the configuration file:
Update the configuration to add JSON parsing:
In the snippet above, you configure a decode_json_fields processor to decode JSON-encoded data in each log entry's message field and attach it to the log event.
Run Filebeat with the updated configuration:
In the output, you will see that all properties in the message field, such as msg, ip, etc., have been added to the log event.
Now that you can parse JSON logs, you will modify attributes on a log event.
Adding and removing fields with Filebeat
The log event contains a sensitive emailAddress field that needs to be protected. In this section, you'll remove the emailAddress field and add a new field to the log event to provide more context.
Update the configuration file:
Update the configuration to add field manipulation:
To modify the log event, you add the drop_fields processor, which has a field option that takes a list of fields to be removed, including the sensitive emailAddress field and the message field. You remove the message field because after parsing the data, properties from the message field were incorporated into the log event, rendering the original message field obsolete.
After writing the code, run the configuration:
Upon running Filebeat, you will notice that the emailAddress field has been successfully removed, and a new env field has been added to the log event:
Now that you can enrich and remove unwanted fields, you will work with conditional statements next.
Working with conditional statements in Filebeat
Filebeat allows you to check a condition and add a field when it evaluates to true. In this section, you will check if the status value equals 200, and if the condition is met, you will add an is_successful field to the log event.
Update the configuration file:
Add conditional field processing:
The when option checks if the status field value equals 200. If true, the is_successful field is added to the log event.
Now, run the configuration:
Filebeat will yield output that looks closely to this:
In the output, the is_successful field has been added to the log entries with an HTTP status code of 200.
That takes care of adding a new field based on a condition.
Redacting sensitive data with Filebeat
Earlier in the article, you removed the emailAddress field to ensure data privacy. However, sensitive fields such as IP addresses and Social Security Numbers (SSN) remain in the log event. Moreover, sensitive data can be inadvertently added to log events by other developers within an organization. Redacting data that match specific patterns allows you to mask any sensitive information without needing to remove entire fields, ensuring the message's significance is preserved.
Update the configuration file to add sensitive data redaction:
Add the following configuration:
In the added code, you define a script written in JavaScript that redacts sensitive information from a log event. The script uses regular expressions to identify SSNs and IP addresses, replacing them with [REDACTED-SSN] and [REDACTED-IP], respectively.
Next, run the configuration:
The log events in the output will now have the IP address and SSN fields redacted.
In scenarios where you have a field like the following:
After processing with Filebeat, only the sensitive portions will be removed, and the log event will appear as follows:
The sensitive portions have now been redacted, and the context of the log message remains intact.
You can now stop the running Docker container by pressing CTRL + C and stop the logify.sh program.
To stop the bash script, obtain the process ID:
Substitute the process ID in the kill command:
The program will now be terminated.
Having successfully redacted sensitive fields, you can now collect logs from Docker containers using Filebeat and centralize them for further analysis and monitoring.
Collecting logs from Docker containers and centralizing logs
In this section, you will containerize the Bash script and use the Nginx hello world Docker image, preconfigured to generate JSON Nginx logs for each incoming request. Subsequently, you will create a Filebeat container to gather logs from both containers and centralize them to Better Stack for analysis.
Dockerizing the Bash script
In this section, you'll containerize the Bash script that generates log data. This step allows you to encapsulate the script and its dependencies.
Navigate back to the logify directory and create a Dockerfile:
In your Dockerfile, add the following code:
In the first line, the latest version of Ubuntu is specified as the base image. Next, the script is copied into the container, made executable, and a directory to store log files is created. Subsequently, any data written to /var/log/logify/app.log is redirected to the standard output so that it can be viewed using the docker logs command. Finally, you specify the command to run when the container is first launched.
Save and exit the file after making these changes. Then, change into the parent project directory:
Next, create a docker-compose.yml file to define the services and volumes for your Docker containers:
Define the Bash script and Nginx services:
In this Docker Compose file, you define two services: logify-script and nginx. The logify-script service is built from the ./logify directory context, creating an image tagged as logify:latest. The nginx service uses the latest version of the nginx-helloworld image and the json-file logging driver for logging purposes. Additionally, port 80 on the host is mapped to port 80 within the container. Ensure no other services use port 80 to prevent conflicts.
To build the logify-script service image and start the containers for each defined service, use the following command:
The -d option puts the services in the background.
Now, check if the services are running:
You should see a "running" status under the "STATUS" column for both containers, which should look like this:
With the containers running, use the curl command to send HTTP requests to the Nginx service:
View all the logs from both containers with the following command:
The output displays logs produced from both containers.
Now that the Bash script and Nginx services are running and generating logs, you can collect and centralize these logs using Filebeat.
Setting up Better Stack source
Before configuring Filebeat to forward logs, let's set up Better Stack to receive them.
Start by creating a free Better Stack account. Once you've logged in, navigate to the Sources section and click the Connect source button:
Give the source a meaningful name like "Logify App Logs", select Docker as the platform, scroll to the bottom, and click Connect Source:
After creating the source, you'll receive a source token (e.g., qU73jvQjZrNFHimZo4miLdxF) and an ingestion host (e.g., s1315908.eu-nbg-2.betterstackdata.com).
Save both—these are required when setting up log forwarding in your Filebeat config:
Keep the token and injection host safe as you'll need them for the Filebeat configuration.
Now you'll configure Filebeat to collect logs from the logify container. Update the filebeat.yml configuration file to collect logs from Docker containers:
Replace the existing configuration with the following, replacing <your_source_token> with your actual source token and <your_ingestion_host> with your ingestion host:
In this configuration, you set up Filebeat's automatic log discovery to collect logs from Docker containers whose image names contain the substring logify. This corresponds to the container defined under the logify-script service. Filebeat uses the container input to read Docker logs specified under paths. Additionally, a processor is added to decode JSON fields.
Now, add the Filebeat service to your docker-compose.yml file:
Update the file to include the Filebeat service:
In this updated configuration, the filebeat-logify service uses the official Filebeat Docker image, with the user set to root. The Filebeat configuration is mounted from the filebeat-config directory.
Start the Filebeat service:
Check that all services are running:
Wait a bit, then revisit Better Stack to verify that Filebeat is forwarding logs. The source should now read “Logs received!”. After that, click Live tail to view your logs in real time:
Navigate to your "Logify logs" source and you should see logs appearing in the live tail:
The logs should show the processed JSON data from your logify application, confirming that Filebeat is successfully collecting and forwarding the logs.
Adding Nginx logs
Now that you have successfully forwarded the Bash script logs, it's time to add Nginx service logs as well. First, create an additional source named "Nginx logs" in Better Stack by following the same steps you did previously. Make sure to copy both the source token and ingestion host to a safe place.
Now you'll need to create a separate Filebeat configuration and service to handle Nginx logs. Create a new configuration file:
Add the following configuration, replacing <your_nginx_source_token> with your actual nginx source token and <your_nginx_ingestion_host> with your nginx ingestion host:
Now add the nginx Filebeat service to your docker-compose.yml:
Add the nginx Filebeat service:
Start the services:
Send a few requests to the Nginx service to generate logs:
Now go back to Better Stack to confirm that the "Nginx logs" source is receiving the logs. Navigate to your "Nginx logs" source and your source will update to show "Logs received!". Click Live tail to view your logs in real-time:
Click on any log entry to expand it and view more detailed information:
Both sources should now be receiving their respective logs, demonstrating successful log centralization with separate sources for different container types.
Monitoring Filebeat health with Better Stack
Filebeat doesn't have a built-in /health endpoint for external monitoring of its instance's health. However, you can configure an endpoint for metrics. Doing so allows you to externally monitor Filebeat to determine whether it's up or down. In this tutorial, you will enable the HTTP endpoint for the filebeat-logify service.
Open the filebeat.yml configuration file:
Add the following lines at the top of the file:
The metric endpoint will be enabled and exposed on port 5066.
Next, update the Docker Compose file to map port 5066 between the host and the container:
Update the filebeat-logify service to expose the metrics port:
Restart the Filebeat service to apply the changes:
Verify that the Filebeat metrics endpoint is functioning properly:
Next, sign into Better Stack uptime monitoring.
On the Monitors page, click the Create monitor button:
Choose the preferred method to trigger Better Stack, provide your server's IP address or domain name on port 5066, and click the Create monitor button:
Better Stack will start monitoring the Filebeat endpoint and provide performance insights:
Now, let's observe how Better Stack responds when Filebeat stops working by stopping all the services:
After some time has passed, return to Better Stack to observe that the status has been updated to "Down":
If you configured Better Stack to send you an email, check your email inbox. You will receive an email alert:
That takes care of monitoring Filebeat using Better Stack.
Final thoughts
In this tutorial, you learned how to use Filebeat with Docker to integrate with Docker containers, Nginx, and Better Stack to manage logs. You started by reading logs from a file and displaying them in a console using Docker containers. Then, you explored various ways to transform log messages using Docker-based Filebeat configurations. After that, you collected logs from multiple Docker containers and forwarded them to Better Stack. Finally, you monitored Filebeat's health using Better Stack and received alerts in case of issues.
As a next step, refer to the Filebeat documentation for more in-depth information. To learn more about Docker and Docker Compose, explore their respective documentation pages: Docker and Docker Compose. To enhance your Docker logging knowledge, check out our comprehensive guide.
Apart from Filebeat, there are other log shippers available that you can explore. Check out the log shippers guide to learn about them.
Thanks for reading, and happy logging!
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.