Pattern detection in logs

Discover patterns in your logs to simplify debugging and monitoring.

The _pattern field is generated automatically for every log entry based on the message content, helping you group similar logs together and identify trends across your application data.

What is the _pattern field?

The _pattern field contains a normalized version of your log message with variable data, like timestamps, IDs, numbers, and other dynamic values, removed. This creates a consistent template that groups similar log entries together.

For example, these different log messages would all share the same pattern: Request to ? returned ? in ?:

Request to /api/search returned 200 in 453.4ms
Request to /api/product returned 404 in 161.6ms  
Request to /api/user returned 500 in 697.2ms

This pattern matching happens automatically — no configuration required 🚀

Using patterns in Live tail

You can use the _pattern field to filter your logs in Live tail.

When viewing logs in Live tail, hover over any log entry and click Filter by similar pattern or Exclude this pattern for a quick filtering.

That will add a new filter into your current query, for example excluding entries that match a specific pattern:

_pattern!=f40f1b38e9b0d68d0ed5a98e516eefda

This is particularly useful when you want to focus on specific types of log entries or exclude noisy patterns from your view.

Managing noise with VRL transformations

High-volume, low-value logs can overwhelm your monitoring and increase costs. Use the _pattern field with VRL transformationss to automatically filter out spam before ingestion.

When viewing logs in Live tail, hover over any log entry and click Mark as spam to set it up quickly.

You'll be prompted to add an extra condition into your VRL transformation.

# Ignore noisy health check logs
# Example: [INFO] Health check - status 200 OK in 34.45ms
if ._pattern == \"6476bd6e601f1d12aa91e82371c88066\" {
    del(.)
}

Since VRL transformations run before log ingestion, filtered events won't count toward your log volume or storage costs.

Analytics and charts

The _pattern field is particularly powerful for log analytics using Explore. You can create charts showing:

• Most common log patterns over time.
• Error pattern distribution across your applications.
• Pattern volume trends to identify unusual activity, with possible anomaly detection alerting.

SELECT 
    count(*) as count,
    sum(length(raw)) as volume,
    _pattern as pattern,
    anyLast(JSONExtractStriing(raw, 'message')) as message_example
FROM {source}
WHERE dt BETWEEN {{start_time}} AND {{end_time}}
GROUP BY pattern
ORDER BY count DESC

This query shows the most frequent log patterns along with example messages, helping you understand what's generating the most log volume.

Under the hood

In summary, this is what Better Stack does to the message contents to generate the patterns:

1. Strips special content.
   Any text within single/double quotes ('...', \"...\") and brackets ([...], (...), {...}) is removed.
2. Filters out variable data.
   Numbers, digits, hex values, UUIDs, timestamps, and trailing punctuation (like :=, ,, ;, ]) are filtered out.
3. Extracts and limits words.
   The algorithm extracts up to the first 100 words from the remaining text. To be included, a word must:
   • Be at least two characters long.
   • Match the pattern ^[a-zA-Z][a-zA-Z._-]*[a-zA-Z]$.
4. Creates a stable pattern.
   The extracted words are joined together to form the pattern template.
5. Calculates an MD5 hash.
   The final pattern is hashed to a 32-character string, which becomes the value of the _pattern field.

Original: [2019-07-24 12:06:21] package.name [DEBUG] got 10 things in 3.1s
Pattern:  package.name got things in
Hash:     6a7b8c9d1e2f3a4b5c6d7e8f9a0b1c2d

The pattern hash in the _pattern field value provides a consistent identifier for grouping and filtering.

Best practices

Start with high-volume patterns: Use the analytics view to identify your most frequent log patterns. These often represent the best candidates for optimization or filtering.

Combine with other fields: Pattern filtering works well when used alongside other log attributes:

_pattern!=6a7b8c9d1e2f3a4b5c6d7e8f9a0b1c2d AND level:ERROR

Monitor pattern changes: New patterns appearing suddenly might indicate new errors or application changes worth investigating.

Use patterns for alerting: Create alerts based on unusual activity in specific patterns rather than trying to parse individual log messages.

The _pattern field transforms log analysis from searching through individual messages to understanding your application's behavioral patterns, making debugging faster and monitoring more effective.