How to Truncate Long Logs in Fluent Bit

To truncate logs in Fluent Bit, the easiest option is to use a Lua script.

Assuming you have the following log entry from an application:

{"status": "200", "ip": "127.0.0.1", "level": 30, "emailAddress": "user@mail.com", "msg": "Task completed successfully", "pid": 2833, "ssn": "407-01-2433", "timestamp": 1696071877}

To truncate the msg field in these logs, you can use a Lua script along with your Fluent Bit configuration file.

Create a Lua script named truncate.lua with the following content:

function truncate_msg(tag, timestamp, record)
    local max_length = 5
    if record.msg and string.len(record.msg) > max_length then
        record.msg = string.sub(record.msg, 1, max_length)
    end
    return 1, timestamp, record
end

The highlighted line specifies the maximum length. Here, it is set to five characters for demonstration purposes. You can set it to a higher value in a real-world application as needed.

Here is how you can reference the Lua script in your Fluent Bit configuration file (fluent-bit.conf):

# Fluent Bit Configuration File

# Service Section
[SERVICE]
    Flush        1
    Daemon       off
    Log_Level    debug
    Parsers_File custom_parsers.conf

# Input Section
[INPUT]
    Name         tail
    Path         /var/log/logify/app.log
    Parser       json_parser
    Tag          filelogs

# Filter Section
[FILTER]
    Name             lua
    Match            *
    Script           truncate.lua
    Call             truncate_msg

# Output Section
[OUTPUT]
    Name         stdout
    Format       json
    Match        *

Adding the filter section allows Fluent Bit to reference the Lua script to truncate the logs.

When you restart Fluent Bit, you will see the logs truncated:

[{"date":1722854611.800805,"status":"200","msg":"Task ","pid":917070,"ssn":"407-01-2433","ip":"127.0.0.1","timestamp":1722854611,"level":30,"emailAddress":"user@mail.com"}]

As you can see, the msg field only contains text with up to 5 characters.

Using this Lua, you can easily control the length of specific log fields in Fluent Bit.

Got an article suggestion? Let us know

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.