Pattern detection in logs

Discover patterns in your logs to simplify debugging and monitoring.

The _pattern field is generated automatically for every log entry based on the message content, helping you group similar logs together and identify trends across your application data.

What is the _pattern field?

The _pattern field contains a normalized version of your log message with variable data - like timestamps, IDs, numbers, and other dynamic values - removed. This creates a consistent template that groups similar log entries together.

For example, these different log messages would all share the same pattern: Request to ? returned ? in ?:

Request to /api/search returned 200 in 453.4ms
Request to /api/product returned 404 in 161.6ms  
Request to /api/user returned 500 in 697.2ms

This pattern matching happens automatically — no configuration required 🚀

Using patterns in Live tail

You can use the _pattern field to filter your logs in Live tail, for example excluding entries that match a specific pattern:

_pattern!=f40f1b38e9b0d68d0ed5a98e516eefda

This is particularly useful when you want to focus on specific types of log entries or exclude noisy patterns from your view.

When viewing logs in Live tail, hover over any log entry and click Filter by similar pattern or Exclude this pattern for a quick filtering.

Managing noise with VRL transformations

High-volume, low-value logs can overwhelm your monitoring and increase costs. Use the _pattern field with VRL transformationss to automatically filter out spam before ingestion:

# Ignore noisy health check logs
# Example: [INFO] Health check - status 200 OK in 34.45ms
if ._pattern == \"6476bd6e601f1d12aa91e82371c88066\" {
    del(.)
}

Since VRL transformations run before log ingestion, filtered events won't count toward your log volume or storage costs.

When viewing logs in Live tail, hover over any log entry and click Mark as spam to set it up quickly.

Analytics and charts

The _pattern field is particularly powerful for log analytics. You can create charts showing:

• Most common log patterns over time.
• Error pattern distribution across your applications.
• Pattern volume trends to identify unusual activity, with possible anomaly detection alerting.

SELECT 
    count(*) as count,
    sum(length(raw)) as volume,
    _pattern as pattern,
    anyLast(getJSON(raw, 'message')) as message_example
FROM {source}
WHERE dt BETWEEN {{start_time}} AND {{end_time}}
GROUP BY pattern
ORDER BY count DESC

This query shows the most frequent log patterns along with example messages, helping you understand what's generating the most log volume.

Under the hood

In summary, this it what Better Stack does to generate the patterns:

1. Removes quoted and bracketed content.
   Any text in quotes \"...\" or '...' and brackets [...], (...), or {...} is stripped out.
2. Filters out variable data.
   Numbers, timestamps, UUIDs, hex values, and other dynamic content are removed.
3. Extracts meaningful words.
   Only alphabetic words meeting minimum length requirements are kept.
4. Creates consistent patterns.
   The remaining words form a stable template that groups similar messages.
5. Calculates MD5 hash.
   This keeps the patterns as short 32-character hexadecimal strings.

Original: [2019-07-24 12:06:21] package.name [DEBUG] got 10 things in 3.1s
Pattern:  package.name got things in
Hash:     6a7b8c9d1e2f3a4b5c6d7e8f9a0b1c2d

The pattern hash in the _pattern field value provides a consistent identifier for grouping and filtering.

Best practices

Start with high-volume patterns - Use the analytics view to identify your most frequent log patterns. These often represent the best candidates for optimization or filtering.

Combine with other fields - Pattern filtering works well when used alongside other log attributes:

_pattern!=6a7b8c9d1e2f3a4b5c6d7e8f9a0b1c2d AND level:ERROR

Monitor pattern changes - New patterns appearing suddenly might indicate new errors or application changes worth investigating.

Use patterns for alerting - Create alerts based on unusual activity in specific patterns rather than trying to parse individual log messages.

The _pattern field transforms log analysis from searching through individual messages to understanding your application's behavioral patterns, making debugging faster and monitoring more effective.