Side note: Ship Fluent Bit logs to Better Stack
Once Fluent Bit is producing structured JSON, you can ship it to Better Stack and get instant search, live tail, and alerting without running your own storage.
Back to Logging guides
How to Collect, Process, and Ship Log Data with Fluent Bit
Stanley Ulili
Updated on January 27, 2026
In distributed systems, efficient log shipping is essential. A log shipper is a tool that gathers logs from various sources, like containers and servers and directs them to a central location for analysis. Several options, including LogStash and Fluentd, are available for this purpose. Among them, Fluent Bit stands out as a lightweight, high-performance log shipper introduced by Treasure Data.
Fluent Bit was developed in response to the growing need for a log shipper that could operate in resource-constrained environments, such as embedded systems and containers. With a minimal memory footprint of 1MB, Fluent Bit efficiently collects logs from multiple sources, transforms the data, and forwards it to diverse destinations for storage and analysis. Key features of Fluent Bit include SQL Stream Processing, backpressure handling, Vendor-Neutral, and Apache 2 Licensed. Fluent Bit also shines with its flexibility because of the pluggable architecture, supporting easy integration and customization. With over 100 built-in plugins, it offers extensive options for collecting, filtering, and forwarding data.
Fluent Bit's reliability is underscored by its adoption by major cloud providers like DigitalOcean, AWS Cloud, and Google Cloud, processing vast amounts of data daily.
In this comprehensive guide, you will use Fluent Bit running in Docker containers to gather logs from diverse sources, transform, and deliver them to various destinations. The tutorial will walk you through reading logs from a file and forwarding them to the console using Docker. Subsequently, you will explore how Fluent Bit can collect logs from multiple containers and route them to a centralized location. Finally, you will monitor Fluent Bit's health to ensure its smooth operation.
Prerequisites
To follow this guide, you need access to a system that has a non-root user account with sudo privileges. You should install Docker and Docker Compose as this tutorial focuses on using Fluent Bit through Docker containers. If you're uncertain about the need for a log shipper, you can read this article on log shippers to understand their benefits, how to choose one, and compare a few options.
Once you've met these prerequisites, create a root project directory that will contain the application and configuration files with the following command:
Move into the newly created directory:
Next, create a subdirectory named logify for the demo application you'll be building in the upcoming section:
Change into the subdirectory:
With these directories in place, you're ready to proceed to the next step, where you'll create the demo logging application.
Developing a demo logging application
In this section, you'll create a sample logging script using Bash that generates log entries at regular intervals and writes them to a file.
Create a logify.sh file within the logify directory. You can use your preferred text editor. This tutorial uses nano:
In your logify.sh file, enter the following contents to generate log entries with Bash:
log-processing-stack/logify/logify.sh
The create_log_entry() function generates log entries in JSON format and includes various details such as HTTP status codes, severity levels, and random log messages. It also intentionally includes sensitive fields like IP address, Social Security Number (SSN), and email address to demonstrate Fluent Bit's ability to remove or redact sensitive data. To learn more about best practices for logging sensitive data, refer to our guide.
Next, the infinite loop continuously invokes the create_log_entry() function to generate a log record every 3 seconds and append them to a specified file in the /var/log/logify/ directory.
When you are finished, save the new changes and make the script executable:
Create a directory to store the application logs:
Assign ownership of the directory to the currently logged-in user:
Then, run the Bash script in the background:
Output
The script will start writing logs to the app.log file. To view the last few log entries, use the tail command:
Output
Each line in the output represents a log event or record.
With the log entries being generated, the next step is to set up Fluent Bit using Docker.
Setting up Fluent Bit with Docker
In this section, you’ll set up the necessary file structure to run Fluent Bit using the official Docker image from fluent/fluent-bit. Running Fluent Bit in Docker offers benefits like easier version control, environment isolation, and consistent deployment.
First, create a directory structure for Fluent Bit configuration files. Navigate back to your project root directory:
Create a directory for Fluent Bit configurations:
Navigate into the configuration directory:
At this point, you’ve created the directory structure needed for Fluent Bit’s configuration files. We won’t add any config files just yet—we’ll come back to this directory shortly when it’s time to set up and run Fluent Bit with Docker.
How Fluent Bit works
Fluent Bit operates as a robust pipeline for handling log data. You can imagine it as a sequence where logs flow through distinct stages, each performing a specific task. Let's break down Fluent Bit's core components and plugins to provide a clearer understanding:
At the beginning of the pipeline, Fluent Bit collects logs from various sources. These logs then pass through a Parser, transforming unstructured data into structured log events. Subsequently, the log event stream encounters the Filter, which can enrich, exclude, or modify the data according to project requirements. After filtration, the logs are temporarily stored in a Buffer, either in memory or the filesystem, ensuring smooth processing. Finally, the Router directs the data to diverse destinations for analysis and storage.
To put this into practice, you can define Fluent Bit's behavior in a configuration file:
Let's look at these components in detail:
[SERVICE]: contains global settings for the running service.[INPUT]: specifies sources of log records for Fluent Bit to collect.[FILTER]: applies transformations to log records.[OUTPUT]: determines the destination where Fluent Bit sends the processed logs.
For these components to do their tasks, they require a plugin. Here is a brief overview of the plugins available for Fluent Bit.
Fluent Bit input plugins
For the [INPUT] component, the following are some of input plugins that can come in handy:
tail: monitors and collects logs from the end of a file, akin to thetail -fcommand.syslog: gathers Syslog logs from a Unix socket server.http: captures logs via a REST endpoint.opentelemetry: fetches telemetry data from OpenTelemetry sources.
Fluent Bit filter plugins
When you need to transform logs, Fluent Bit provides a range of filter plugins suited for different modifications:
record_modifier: modifies log records.lua: alters log records using Lua scripts.grep: matches or excludes log records, similar to thegrepcommand.modify: changes log records based on specified conditions or rules.
Fluent Bit output plugins
To dispatch logs to various destinations, Fluent Bit offers versatile output plugins:
file: write logs to a specified file.amazon_s3: sends logs, metrics to Amazon S3.http: pushes records to an HTTP endpoint.websocket: forwards log records to a WebSocket endpoint.
Now that you have a rough idea of how Fluent Bit works, you can proceed to the next section to start using Fluent Bit with Docker.
Getting started with Fluent Bit using Docker
In this section, you will configure Fluent Bit to read logs from a file using the tail input plugin and display them in the console. You'll run Fluent Bit in a Docker container with properly mounted configuration files.
First, create the Fluent Bit configuration file:
Add the following configuration code:
fluent-bit-config/fluent-bit.conf
The [SERVICE] defines global settings for Fluent Bit. It specifies that Fluent Bit should flush every 1 second, run in the foreground, and set the log level to debug.
The [INPUT] uses the tail plugin to read logs from the specified file at /var/log/logify/app.log. The Tag allows other Fluent Bit components, such as [FILTER] and [OUTPUT], to identify these log records.
The [OUTPUT] component uses the stdout plugin to forward logs to the console. The Match parameter ensures only logs with the filelogs tag are delivered to the console.
After making these changes, save the file.
Next, validate your configuration file for errors using Docker:
Output
If the output displays "configuration test is successful", your configuration file is valid and error-free.
Make sure your Bash program is still running in the background. If not, restart it.
Now, start Fluent Bit using Docker with the proper volume mounts:
The command mounts:
- Your configuration directory (
$(pwd)) to/fluent-bit/etcinside the container - The log directory (
/var/log/logify) to the same path inside the container
When Fluent Bit starts, you should see an output similar to the following:
Output
Following that, you will see the log messages appear:
Output
Fluent Bit is now displaying the log messages along with additional context. You can exit Fluent Bit by pressing CTRL + C.
Transforming logs with Fluent Bit
When collecting logs with Fluent Bit, processing them to enhance their utility is often necessary. Fluent Bit provides a powerful array of filter plugins designed to transform event streams effectively. In this section, we will explore various essential log transformation tasks:
- Parsing JSON logs.
- Removing unwanted fields.
- Adding new fields.
- Converting Unix timestamps to the ISO format.
- Masking sensitive data.
Parsing JSON logs with Fluent Bit
When working with logs generated in JSON format, it's crucial to parse them accurately. This ensures the data maintains its integrity and adheres to the expected structure. This section will focus on parsing JSON log records as valid JSON to provide a well-defined structure.
To do that, let's examine a log event from the last section in detail:
Output
Upon close inspection, you will see that Fluent Bit adds key=value pairs, and the data here needs a consistent JSON structure.
You can create a Parser to parse logs as JSON in Fluent Bit.
In your text editor, create a parser_json.conf file:
In your parser_json.conf file, add the following code:
fluent-bit-config/parser_json.conf
The [PARSER] component takes the parser's name and the format in which log events should be parsed, which is json here.
In the Fluent Bit configuration file fluent-bit.conf, make the following modifications:
fluent-bit-config/fluent-bit.conf
The Parsers_File parameter references the parser_json.conf file, which defines the json_parser for parsing JSON logs.
In the [INPUT] component, you add the Parser parameter with the value json_parser. This specifies that the incoming logs should be parsed using the JSON parser defined in parser_json.conf.
Finally, in the [OUTPUT] section, you set the format parameter to json, ensuring that the logs forwarded to the output are in the JSON format.
After making these changes, save the configuration file and restart Fluent Bit using Docker:
Output
You can now observe that the logs are formatted in the JSON format.
Now, you can stop Fluent Bit with CTRL + C.
You have learned how to parse incoming JSON logs correctly. Fluent Bit provides various parsers to handle diverse log formats:
regex: uses regular expressions to parse log eventslogfmt: parses log records which are in the Logfmt format.lstv: parse log events in the LSTV format format.
These parsing methods offer flexibility, allowing Fluent Bit to handle many log formats efficiently.
Now that you can parse the JSON logs, you will alter the log records attribute in the next section.
Adding and removing fields with Fluent Bit
In this section, you'll customize log records by removing sensitive data and adding new fields. Precisely, you will remove the emailAddress field due to its sensitive nature and add a hostname field to enhance log context.
Open your Fluent Bit configuration file in your text editor:
Integrate the following [FILTER] component into your configuration:
fluent-bit-config/fluent-bit.conf
In the [FILTER] component, the name parameter denotes that the record_modifier plugin is being used. To exclude the emailAddress field, you use the Remove_key parameter. The Record parameter also introduces a new field called hostname, which is automatically populated with the system's hostname information.
Save your changes and restart Fluent Bit using Docker to apply the modifications:
When Fluent Bit runs, you will observe the log events without the emailAddress field, and the hostname field will be incorporated into the log events:
Output
That takes care of removing fields and adding new fields. In the next section, you will format the timestamps.
Formatting dates with Fluent Bit
The Bash generates logs with a Unix timestamp, representing the number of seconds that elapsed since January 1st, 1970, at 00:00:00 UTC. While these timestamps are precise, they aren't user-friendly. As a result, you'll convert them into the more human-readable ISO format.
At the time of writing, it isn't easy to do this with existing plugins. A better option is to use a Lua script to perform the conversion and reference it in the configuration file using the lua plugin.
Create the convert_timestamp.lua file:
Next, add the following code to convert the timestamp field from Unix timestamp to ISO format:
fluent-bit-config/convert_timestamp.lua
The append_converted_timestamp() function creates a new record and sets the timestamp field to the value returned by the os.date() method, configured to format dates into the ISO format.
Save and exit your file. Open the Fluent Bit configuration:
Update the configuration to include the Lua script in the [FILTER] component:
fluent-bit-config/fluent-bit.conf
The [FILTER] component uses the lua plugin to modify log records dynamically. The Script parameter holds the path to the Lua script file. Meanwhile, the Call parameter specifies the function within the Lua script that will be invoked to perform the conversion.
Upon saving the file, start Fluent Bit using Docker:
Fluent Bit will yield output similar to the following:
Output
The timestamp field is now in a human-readable ISO format. This change will improve the readability of your logs to understand when they occur.
Working with conditional statements in Fluent Bit
While Fluent Bit doesn't natively support conditional statements, you can achieve similar functionality by leveraging the modify plugin. In this section, you'll learn how to check if the status field equals 200 and add an is_successful field set to true when this condition is met.
First, open your fluent-bit.conf configuration file:
Inside the file, add the following [FILTER] component:
fluent-bit-config/fluent-bit.conf
The modify plugin provides the Condition parameter with a Key_Value_Equals option that checks if the status field value equals "200". If the condition is met, the Add option appends an is_successful field to the log event.
Save the configuration file and start Fluent Bit using Docker:
Output
You will now see the is_successful field, indicating the outcomes where the status field equals 200.
Masking sensitive data with Fluent Bit
In the earlier steps, you successfully removed the emailAddress from the log records, yet sensitive fields like IP addresses and Social Security Numbers remain. For personal information safety in the logs, it's crucial to mask this data. This becomes especially pertinent when sensitive details are part of a field that can't be entirely removed.
While many built-in plugins redact entire value fields, using a Lua script is the best solution since you can easily specify and selectively mask specific data portions.
Create a redact.lua script with your text editor:
Add the following code to the redact.lua script:
fluent-bit-config/redact.lua
In this code snippet, the redact_sensitive_portions() function iterates through each field, using the string.gsub() method to locate and replace IP addresses and Social Security Numbers with the text "REDACTED".
The filter() function acts as the entry point. It calls the redact_sensitive_portions function to mask sensitive portions within the log record. After processing, the modified record is returned through the filter object.
Now, open your Fluent Bit configuration file:
Add the [FILTER] component to reference the redact.lua script:
fluent-bit-config/fluent-bit.conf
The [FILTER] component references the redact.lua file, and the CALL parameter invokes the filter function as the entry point.
When you are done, start Fluent Bit using Docker:
Output
The IP address and SSN have now been masked. In scenarios where a field contains both the IP address and SSN like this:
Output
Fluent Bit will redact the sensitive portions only:
Output
Let's now stop the logify.sh. To do that, you will need program's process ID:
Output
Then, terminate the program with the kill command and ensure the process ID has been substituted.
Now that you can mask sensitive portions, you can move on to collecting logs from docker containers.
Collecting logs from Docker containers and centralizing logs
In this section, you'll containerize the Bash program and use an Nginx hello world Docker image, which has been preconfigured to generate JSON Nginx logs upon each incoming request. Subsequently, you will deploy a Fluent Bit container to collect logs from Bash and Nginx containers and forward them to Better Stack for centralization.
Dockerizing the Bash script
Containerization lets you encapsulate the script and its dependencies, which makes it portable across different environments.
To containerize the Bash program, navigate back to the logify directory:
Create a Dockerfile, which will contain instructions on how to build the image:
In your Dockerfile, add the following lines of code:
log-processing-stack/logify/Dockerfile
In this Dockerfile, you start with the recent version of Ubuntu as the base image. You then copy the script into the container, make it executable, and create a directory where the application will write the logs. You then redirect all the log data written to /var/log/logify/app.log to the standard output. And finally, you specify the command to run when the container starts.
Now, move into the parent project directory:
Create a docker-compose.yml file to orchestrate multiple containers:
First, let's start with a basic configuration that includes just the logify service and Fluent Bit. This progressive approach ensures you can verify the first source works before adding others:
log-processing-stack/docker-compose.yml
In this configuration, the logify-script service is linked to the fluent-bit service and depends on it. The service is configured to use the fluentd driver for logging, and the fluentd-address specifies the address to which Docker will send the logs. The tag docker.logify helps identify the source of the Docker logs.
The fluent-bit service uses the pre-built fluent/fluent-bit image and incorporates volume mappings for Fluent Bit's configuration files. The command parameter specifies the execution of fluent-bit.conf when the container starts. Additionally, port 24224 is exposed to receive logs from other containers.
Now, update the Fluent Bit configuration to handle Docker logs. Navigate to the fluent-bit-config directory:
Create a new configuration file for Docker log collection:
Replace the existing configuration with the following:
log-processing-stack/fluent-bit-config/fluent-bit.conf
This configuration uses the forward plugin to receive logs sent by Docker containers through port 24224. For now, we're outputting to stdout to verify the setup works.
Navigate back to the project root and build the containers:
Check that the containers are running:
You should see both containers running:
Output
View the Fluent Bit logs to see if it's collecting the logify container logs:
You should see logs from the logify container being processed by Fluent Bit:
Output
Centralizing logs to Better Stack
Now that you've verified the basic setup works, let's centralize the logs to Better Stack.
If you want the simplest path, watch the quick walkthrough below. It shows how to set up a Better Stack collector and start shipping logs from a Docker or Kubernetes environment in minutes, with sensible defaults for batching, compression, and sampling:
If you prefer to keep using your existing Fluent Bit pipeline, skip the video and follow the step by step instructions in this section.
First, create a free Better Stack account. When you are logged in, visit the Sources section:
Create a new source by clicking Connect source. Give it a clear name, like "Logify App Logs", and select Docker as the platform. Then scroll down and click Connect Source to finish setup:
Once the source is created, copy both the Source Token and Ingesting Host fields to the clipboard:
If you want a quick overview of how sources work in Better Stack, watch the short video below.
Update the Fluent Bit configuration to send logs to Better Stack. Open the configuration file:
Update the [OUTPUT] section:
log-processing-stack/fluent-bit-config/fluent-bit.conf
Replace <your_logify_source_token> with the source token and <your_ingesting_host> with the ingesting host you copied from Better Stack. You can find both values in your source configuration.
Restart the services:
Once the fluent bit container is running, head back to the Better Stack source page and scroll to the Verify data collection section. After a short wait, you should see a Logs received! message—this confirms that logs from your Bash script container are being delivered to Better Stack:
Check Better Stack to verify that the log entries are being successfully delivered. You should see the log entries uploading to Better Stack's interface in the live tail view:
Select any log entry to see more detailed information and metadata:
If you want a quick tour of Live Tail and how to drill into individual log events, watch the walkthrough below:
Adding Nginx container logs
Now that the first source is working correctly, let's add the Nginx container. First, create a new source for Nginx logs on Better Stack following the same steps as before, but name it "Nginx logs" and select "Fluent-bit" as the platform.
After creating the Nginx source, copy its source token and update the docker-compose.yml file to include the Nginx service:
Add the Nginx service to your configuration:
log-processing-stack/docker-compose.yml
Now update the Fluent Bit configuration to handle both sources:
Add a separate output for Nginx logs:
log-processing-stack/fluent-bit-config/fluent-bit.conf
Replace <your_nginx_source_token> with the Nginx source token and <your_ingesting_host> with the ingesting host from Better Stack.
Restart all services:
Generate some Nginx logs by sending HTTP requests:
The Nginx logs will be uploaded to Better Stack:
Monitoring Fluent Bit health with Better Stack
Fluent Bit provides a health endpoint that allows you to monitor Fluent Bit's health using external tools like Better Stack. The health endpoint is already enabled in your configuration through the [SERVICE] section:
These configurations instruct Fluent Bit to start listening for requests on port 2020 to check its health status. Port 2020 is already exposed in your Docker Compose configuration.
Next, update the docker-compose.yml file to expose the port that hosts the health endpoint:
log-processing-stack/docker-compose.yml
Now, start the Fluent Bit service with the updated changes:
Verify the health endpoint is functioning:
Output
If you prefer the quickest path, watch the short walkthrough below. It shows how to create an Uptime monitor for a health check endpoint:
Next, log in to Better Stack. On the Monitors page, click the Create monitor button:
Then, select the suitable triggering option in Better Stack, your preferred notification preferences and input your server's IP address or domain name, followed by the /api/v1/health endpoint on port 2020. After that, click the Create monitor button:
Upon completion, Better Stack will regularly monitor Fluent Bit's health endpoint:
Let's see what happens when Fluent Bit malfunctions. Halt all services using the command:
After a brief interval, check Better Stack. The status will transition to "Down":
When there is an outage, Better Stack will promptly notify you. An email alert will be dispatched detailing the downtime, allowing you to proactively manage Fluent Bit's health and swiftly address the problem:
With these tools, you can proactively manage Fluent Bit's health and swiftly respond to operational interruptions.
Side note: Visualize and explore your logs in Better Stack
Once your logs are flowing, you can go beyond search and use Better Stack to visualize patterns over time.
Final thoughts
In this comprehensive article, you learned how Fluent Bit can be integrated with Docker, Nginx, and Better Stack for managing logs using a containerized approach. You started by running Fluent Bit in Docker containers to read logs from files and display them in the output. You then employed Fluent Bit to collect logs from multiple Docker containers and centralize them on Better Stack using a progressive setup that verified each source before adding others. Finally, you set up a health endpoint to monitor Fluent Bit's health using Better Stack.
You can now effectively manage logs on your system using Fluent Bit running in Docker containers. To delve deeper into Fluent Bit's capabilities, consult the documentation. Fluent Bit offers powerful features such as SQL stream processing, which you can explore further here. Additionally, to hone your skills in Docker and Docker Compose, refer to their respective documentation pages: Docker and Docker Compose. To gain insights into Docker logging, consult our comprehensive guide.
Thanks for reading, and happy logging!
Got an article suggestion?
Let us know
Next article
How to Collect, Process, and Ship Log Data with Logstash
Learn how Logstash streamlines the collection, processing, and shipping of log data at scale, boosting observability and troubleshooting capabilities.
→
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.