# Pattern detection in logs

**Discover patterns in your logs to simplify debugging and monitoring.**

The `_pattern` field is generated automatically for every log entry based on the message content, helping you group similar logs together and identify trends across your application data.

![Pattern filtering in Live tail](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/5c06fe2f-463f-4c5d-52af-fb4209851400/lg2x =2050x1282)
## What is the `_pattern` field?

The `_pattern` field contains a normalized version of your log message with variable data, like timestamps, IDs, numbers, and other dynamic values, removed. This creates a consistent template that groups similar log entries together.

For example, these different log messages would all share the same pattern: `Request to ? returned ? in ?`:

```text
[label Logs sharing a pattern]
Request to /api/search returned 200 in 453.4ms
Request to /api/product returned 404 in 161.6ms  
Request to /api/user returned 500 in 697.2ms
```

This pattern matching happens automatically — no configuration required 🚀

[note]
#### Can't see the `_pattern` field in your logs?

A pattern might not be generated if the log does not contain a `message` field at all, or if the `message` is empty or not a text.
[/note]

## Using patterns in Live tail

You can use the `_pattern` field to filter your logs in Live tail.

When viewing logs in [Live tail](https://telemetry.betterstack.com/team/0/tail ";_blank"), hover over any log entry and click **Filter by similar pattern** or **Exclude this pattern** for a quick filtering.

![Pattern-based filtering](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/3030f031-936e-49cf-fe57-fb7858364d00/md1x =2034x848)

That will add a new filter into your current query, for example excluding entries that match a specific pattern:

```text
[label Live tail query excluding a pattern]
_pattern!=f40f1b38e9b0d68d0ed5a98e516eefda
```

This is particularly useful when you want to focus on specific types of log entries or exclude noisy patterns from your view.

## Managing noise with VRL transformations

High-volume, low-value logs can overwhelm your monitoring and increase costs.
Use the `_pattern` field with [VRL transformations](https://betterstack.com/docs/logs/using-logtail/transforming-ingested-data/logs-vrl/)s to automatically filter out spam before ingestion.

When viewing logs in [Live tail](https://telemetry.betterstack.com/team/0/tail ";_blank"), hover over any log entry and click **Mark as spam** to set it up quickly.

![Mark as spam using patterns](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/1a882ce9-a92b-4b3f-5565-7f6003c37900/md1x =1874x920)

You'll be prompted to add an extra condition into your VRL transformation.

```vrl
[label Exclude spam using VRL transformation]
# Ignore noisy health check logs
# Example: [INFO] Health check - status 200 OK in 34.45ms
if ._pattern == \"6476bd6e601f1d12aa91e82371c88066\" {
    del(.)
}
```

Since [VRL transformations](https://betterstack.com/docs/logs/using-logtail/transforming-ingested-data/logs-vrl/) run before log ingestion, filtered events won't count toward your log volume or storage costs.

## Analytics and charts

The `_pattern` field is particularly powerful for log analytics using [Explore](https://betterstack.com/docs/logs/using-logtail/explore-logs/).
You can create charts showing:

- **Most common log patterns** over time.
- **Error pattern distribution** across your applications.
- **Pattern volume trends** to identify unusual activity, with possible [anomaly detection alerting](https://betterstack.com/docs/logs/dashboards/alerts/).

```sql
[label Explore used log patterns using SQL]
SELECT 
    count(*) as count,
    sum(length(raw)) as volume,
    _pattern as pattern,
    anyLast(JSONExtractStriing(raw, 'message')) as message_example
FROM {source}
WHERE dt BETWEEN {{start_time}} AND {{end_time}}
GROUP BY pattern
ORDER BY count DESC
```

This query shows the most frequent log patterns along with example messages, helping you understand what's generating the most log volume.

![Pattern analytics chart](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/5bfd2d2b-9172-4032-2951-26eba09ba900/md2x =1786x1438)

## Under the hood

In summary, this is what Better Stack does to the `message` contents to generate the patterns:

1.  **Strips special content.**  
    Any text within single/double quotes (`'...'`, `\"...\"`) and brackets (`[...]`, `(...)`, `{...}`) is removed.
2.  **Filters out variable data.**  
    Numbers, digits, hex values, UUIDs, timestamps, and trailing punctuation (like `:=`, `,`, `;`, `]`) are filtered out.
3.  **Extracts and limits words.**  
    The algorithm extracts up to the first 100 words from the remaining text. To be included, a word must:
    - Be at least two characters long.
    - Match the pattern `^[a-zA-Z][a-zA-Z._-]*[a-zA-Z]$`.
4.  **Creates a stable pattern.**  
    The extracted words are joined together to form the pattern template.
5.  **Calculates an MD5 hash.**  
    The final pattern is hashed to a 32-character string, which becomes the value of the `_pattern` field.

```text
[label Pattern detection algorithm]
Original: [2019-07-24 12:06:21] package.name [DEBUG] got 10 things in 3.1s
Pattern:  package.name got things in
Hash:     6a7b8c9d1e2f3a4b5c6d7e8f9a0b1c2d
```

The pattern hash in the `_pattern` field value provides a consistent identifier for grouping and filtering.

## Best practices

**Start with high-volume patterns:** Use the analytics view to identify your most frequent log patterns. These often represent the best candidates for optimization or filtering.

**Combine with other fields:** Pattern filtering works well when used alongside other log attributes:

```text
[label Filtering errors except ones matching a pattern]
_pattern!=6a7b8c9d1e2f3a4b5c6d7e8f9a0b1c2d AND level:ERROR
```

**Monitor pattern changes:** New patterns appearing suddenly might indicate new errors or application changes worth investigating.

**Use patterns for alerting:** Create alerts based on unusual activity in specific patterns rather than trying to parse individual log messages.

The `_pattern` field transforms log analysis from searching through individual messages to understanding your application's behavioral patterns, making debugging faster and monitoring more effective.