Transforming ingested logs with VRL

Transform your log data in real time with VRL in Better Stack. VRL is a lightweight, expressive scripting language designed for log transformations. It allows you to modify, enrich, or filter log data before ingestion.

For complete details on VRL syntax and capabilities, please refer to the official VRL documentation.

Getting started

To apply VRL transformations in Better Stack, navigate to Sources → your source → Configure → Transformations tab.

In the editor, you can write VRL scripts that modify each log event as it’s received. For example, you can use this to rename fields, parse data, filter you logs before ingesting, or redact sensitive information.

Examples

Removing unwanted fields

Remove unwanted fields to reduce noise or protect sensitive data.

del(.password)
del(.context.referer)

Anonymizing logs, redacting sensitive data (PII)

Redact sensitive information using regex patterns. This transformation will remove any emails or IP addresses from your logs before they are saved in Better Stack.

email_pattern = r'[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}'
ipv4_pattern  = r'\b(?:\d{1,3}\.){3}\d{1,3}\b'
ipv6_pattern  = r'\b(?:(?:[A-Fa-f0-9]{1,4}:){7}[A-Fa-f0-9]{1,4}|(?:[A-Fa-f0-9]{1,4}:){1,7}:|:(?::[A-Fa-f0-9]{1,4}){1,7})\b'
. = redact(., filters: [email_pattern, ipv4_pattern, ipv6_pattern])

Dropping unnecessary logs

Filter out log events that aren’t needed, for example debug-level logs. Since VRL transformations happen before the logs are ingested, you can use them to reduce your log usage.

if .level == "debug" {
    del(.)
}

Parsing structured data

You can parse structured data out of your logs, either by using built-in parsing functions or custom regexes.

# Example message: "path=/ host=example.com response_time=1.23s status=200"
.parsed = parse_logfmt(.message) ?? {}

time_format = r'(?P<number>\d+(?:.\d+)?)(?P<unit>[a-z]+)'
response_time = parse_regex(.parsed.response_time, time_format) ?? {}

if (response_time.unit == "ms") {
    .parsed.response_time_ms = parse_float!(response_time.number)
} else if (response_time.unit == "s") {
    .parsed.response_time_ms = parse_float!(response_time.number) * 1000
}

Tips when writing VRL transformations

VRL transformations are executed before logs are ingested. This means that any removed fields or dropped events never get stored, reducing log volume, and optimizing storage costs.

Start small, test your VRL scripts, and refine them over time to suit your needs. With VRL, you gain powerful, real-time control over your log data right at the ingestion point.

Using the Test transformation button with a sample of your logs used as input can give you quick feedback.

The official VRL documentation is a great place to learn about available functions, error handling, or more details about the syntax.

You can tweak and test your regexes using regex101.com, selecting the Golang flavor for consistent results.

VRL transformation and JSON

When sending strings containing JSON data, Better Stack will automatically parse the JSON into structured data.

The VRL transformation happens before the JSON is parsed. If you want to transform fields contained in the structured data, you might need to parse it manually inside your transformation.

message_json = parse_json(.message) ?? {}
if message_json != {} {
    .message = message_json
    del(.message.my_field)
}