Transforming data with VRL

Transform your data in real time with VRL in Better Stack. VRL is a lightweight, expressive scripting language designed for data transformations. It allows you to modify, enrich, or filter your data before ingestion.

For complete details on VRL syntax and capabilities, please refer to the official VRL documentation.

Getting started

To apply VRL transformations in Better Stack, navigate to Sources → your source → Configure → Transformations tab.

In the editor, you can write VRL scripts that modify each event as it’s received. For example, you can use this to rename fields, parse data, filter your data before ingesting, or redact sensitive information.

Examples

Removing unwanted fields

Remove unwanted fields to reduce noise or protect sensitive data.

del(.password)
del(.context.referer)

Anonymizing data, redacting sensitive data (PII)

Redact sensitive information using regex patterns. This transformation will remove any emails or IP addresses from your data before they are saved in Better Stack.

email_pattern = r'[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}'
ipv4_pattern  = r'\b(?:\d{1,3}\.){3}\d{1,3}\b'
ipv6_pattern  = r'\b(?:(?:[A-Fa-f0-9]{1,4}:){7}[A-Fa-f0-9]{1,4}|(?:[A-Fa-f0-9]{1,4}:){1,7}:|:(?::[A-Fa-f0-9]{1,4}){1,7})\b'
. = redact(., filters: [email_pattern, ipv4_pattern, ipv6_pattern])

Dropping unnecessary events

Filter out events that aren’t needed, for example debug-level logs. Since VRL transformations happen before the data is ingested, you can use them to reduce your usage.

if .level == "debug" {
    del(.)
}

Splitting one event into many

You can split a single record into multiple records. This is useful when one record contains a batch of events.

lines = split(.message, "\n") ?? []

# Create an array to hold the new log events
new_events = []

# Iterate over each line and create a new event
for_each(lines) -> |index, line| {
  event = .
  event.message = line
  new_events = push(new_events, event)
}

# Replace the original event with the array of new events
if (new_events != []) {
    . = new_events
}

{
  "message": "report 1: success\nreport 2: failure",
  "source_component": "reporter"
}

{
  "message": "report 1: success",
  "source_component": "reporter"
}
{
  "message": "report 2: failure",
  "source_component": "reporter"
}

Parsing structured data

You can parse structured data out of your events, either by using built-in parsing functions or custom regexes.

# Example message: "path=/ host=example.com response_time=1.23s status=200"
.parsed = parse_logfmt(.message) ?? {}

time_format = r'(?P<number>\d+(?:.\d+)?)(?P<unit>[a-z]+)'
response_time = parse_regex(.parsed.response_time, time_format) ?? {}

if (response_time.unit == "ms") {
    .parsed.response_time_ms = parse_float!(response_time.number)
} else if (response_time.unit == "s") {
    .parsed.response_time_ms = parse_float!(response_time.number) * 1000
}

Tips when writing VRL transformations

VRL transformations are executed before data is ingested. This means that any removed fields or dropped events never get stored, reducing volume, and optimizing storage costs.

Start small, test your VRL scripts, and refine them over time to suit your needs. With VRL, you gain powerful, real-time control over your data right at the ingestion point.

Using the Test transformation button with a sample of your data used as input can give you quick feedback.

The official VRL documentation is a great place to learn about available functions, error handling, or more details about the syntax.

You can tweak and test your regexes using regex101.com, selecting the Golang flavor for consistent results.

Handling errors in VRL

Using ! after a function in your transformation can lead to runtime errors.

# This transformation will fail if .message is a non-numeric string
.parsed_number = to_int!(.message)

If that happens, Better Stack will not drop the event. You still can find the untransformed event in Logs & traces, including extra fields _ingest.error and _ingest.transform_error so you can identify what went wrong.

{
  # your untransformed event...
  "_ingest.error": "transformation failed",
  "_ingest.transform_error": "ERROR: Runtime error: function call error for to_int at (17:34): Invalid integer \"Request to /api/user returned 200 in 492.8ms\": invalid digit found in string"
}

To avoid transformation failures, use Vector error handling to safely handle expected edge cases:

# When to_int fails, another expression can be used, e.g. null
.parsed_number = to_int(.message) ?? null

# You can assign both result and error, and handle them separately
parsed_number, err = to_int(.message)
if (err == null) {
  .parsed_number = parsed_number
} else {
  .parsed_number_error = err
}

This way, you can ensure a reliable transformation! 🚀

VRL transformation and JSON

When sending strings containing JSON data, Better Stack will automatically parse the JSON into structured data.

The VRL transformation happens after the JSON is parsed. Feel free to use VRL transformations to rename the extracted fields.

# Assuming the original message was a JSON string:
# '{ "message": "my message", "my_field": 123, ... }'

text_message = del(.message.message)
.extra = .message
.message = text_message