# Transforming ingested logs with VRL
**Transform your log data in real time with VRL in Better Stack.** VRL is a lightweight, expressive scripting language designed for log transformations. It allows you to modify, enrich, or filter log data before ingestion.
For complete details on VRL syntax and capabilities, please refer to the [official VRL documentation](https://vector.dev/docs/reference/vrl/).
## Getting started
To apply VRL transformations in Better Stack, navigate to [Sources](https://telemetry.betterstack.com/team/0/sources ";_blank") → **your source** → **Configure** → **Transformations** tab.
In the editor, you can write VRL scripts that modify each log event as it’s received. For example, you can use this to rename fields, parse data, filter you logs before ingesting, or redact sensitive information.
## Examples
### Removing unwanted fields
Remove unwanted fields to reduce noise or protect sensitive data.
```vrl
[label Removing unwanted fields]
del(.password)
del(.context.referer)
```
### Anonymizing logs, redacting sensitive data (PII)
Redact sensitive information using regex patterns. This transformation will remove any emails or IP addresses from your logs before they are saved in Better Stack.
```vrl
[label Redacting sensitive data]
email_pattern = r'[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}'
ipv4_pattern = r'\b(?:\d{1,3}\.){3}\d{1,3}\b'
ipv6_pattern = r'\b(?:(?:[A-Fa-f0-9]{1,4}:){7}[A-Fa-f0-9]{1,4}|(?:[A-Fa-f0-9]{1,4}:){1,7}:|:(?::[A-Fa-f0-9]{1,4}){1,7})\b'
. = redact(., filters: [email_pattern, ipv4_pattern, ipv6_pattern])
```
### Dropping unnecessary logs
Filter out log events that aren’t needed, for example debug-level logs. Since VRL transformations happen before the logs are ingested, you can use them to reduce your log usage.
```vrl
[label Dropping unnecessary logs]
if .level == "debug" {
del(.)
}
```
[note]
**Want to drop similar looking messages?**
In [Logs & traces](https://telemetry.betterstack.com/team/0/tail ";_blank"), hover over any log entry and click **Mark as spam** to set it up quickly. Learn more about [detecting log patterns](https://betterstack.com/docs/logs/using-logtail/using-pattern-field/).
[/note]
### Splitting one log into many
You can split a single log record into multiple records. This is useful when one log entry contains a batch of events.
[code-tabs]
```vrl
[label Split the message by newline]
lines = split(.message, "\n") ?? []
# Create an array to hold the new log events
new_events = []
# Iterate over each line and create a new event
for_each(lines) -> |index, line| {
event = .
event.message = line
new_events = push(new_events, event)
}
# Replace the original event with the array of new events
if (new_events != []) {
. = new_events
}
```
```json
[label Example input]
{
"message": "report 1: success\nreport 2: failure",
"source_component": "reporter"
}
```
```json
[label Example output]
{
"message": "report 1: success",
"source_component": "reporter"
}
{
"message": "report 2: failure",
"source_component": "reporter"
}
```
[/code-tabs]
### Parsing structured data
You can parse structured data out of your logs, either by using [built-in parsing functions](https://vector.dev/docs/reference/vrl/functions/#parse-functions) or custom regexes.
```vrl
[label Parse using logfmt format and regex]
# Example message: "path=/ host=example.com response_time=1.23s status=200"
.parsed = parse_logfmt(.message) ?? {}
time_format = r'(?P\d+(?:.\d+)?)(?P[a-z]+)'
response_time = parse_regex(.parsed.response_time, time_format) ?? {}
if (response_time.unit == "ms") {
.parsed.response_time_ms = parse_float!(response_time.number)
} else if (response_time.unit == "s") {
.parsed.response_time_ms = parse_float!(response_time.number) * 1000
}
```
## Tips when writing VRL transformations
**VRL transformations are executed before logs are ingested.** This means that any removed fields or dropped events never get stored, reducing log volume, and optimizing storage costs.
Start small, test your VRL scripts, and refine them over time to suit your needs. With VRL, you gain powerful, real-time control over your log data right at the ingestion point.
Using the **Test transformation** button with a sample of your logs used as input can give you quick feedback.
The [official VRL documentation](https://vector.dev/docs/reference/vrl/) is a great place to learn about [available functions](https://vector.dev/docs/reference/vrl/functions/), [error handling](https://vector.dev/docs/reference/vrl/errors/), or more details about the syntax.
You can tweak and test your regexes using [regex101.com](https://regex101.com), selecting the **Golang flavor** for consistent results.
## Handling errors in VRL
Using `!` after a function in your transformation can lead to runtime errors.
```vrl
[label Fallible VRL transformation]
# This transformation will fail if .message is a non-numeric string
.parsed_number = to_int!(.message)
```
If that happens, Better Stack will not drop the event. You still can find the untransformed event in [Logs & traces](https://betterstack.com/docs/logs/using-logtail/live-tail-query-language/), including extra fields `_ingest.error` and `_ingest.transform_error` so you can identify what went wrong.
```json
[label Transformation error metadata in Logs & traces]
{
# your untransformed event...
"_ingest.error": "transformation failed",
"_ingest.transform_error": "ERROR: Runtime error: function call error for to_int at (17:34): Invalid integer \"Request to /api/user returned 200 in 492.8ms\": invalid digit found in string"
}
```
To avoid transformation failures, use [Vector error handling](https://vector.dev/docs/reference/vrl/errors/#runtime-errors) to safely handle expected edge cases:
[code-tabs]
```vrl
[label Coalescing errors]
# When to_int fails, another expression can be used, e.g. null
.parsed_number = to_int(.message) ?? null
```
```vrl
[label Assigning errors]
# You can assign both result and error, and handle them separately
parsed_number, err = to_int(.message)
if (err == null) {
.parsed_number = parsed_number
} else {
.parsed_number_error = err
}
```
[/code-tabs]
This way, you can ensure a reliable transformation! 🚀
## VRL transformation and JSON
When sending strings containing JSON data, [Better Stack will automatically parse the JSON](https://betterstack.com/docs/logs/ingesting-data/http/logs/string-and-json-format/) into structured data.
**The VRL transformation happens after the JSON is parsed.** Feel free to use VRL transformations to rename the extracted fields.
```vrl
[label Renaming fields in JSON-formatted message]
# Assuming the original message was a JSON string:
# '{ "message": "my message", "my_field": 123, ... }'
text_message = del(.message.message)
.extra = .message
.message = text_message
```