How to Fix The “Pattern Not Match” Error in Fluentd

Fluentd Logging
Better Stack Team
Updated on February 5, 2024

If you're new to FluentD and looking to build a solid foundation, consider checking out our comprehensive guide on how to collect, process, and ship log data with Fluentd. It provides valuable insights and knowledge for effectively using Fluentd.

Now, addressing the issue of the "pattern not matched" error in Fluentd, let's assume you have the following configuration to parse JSON logs:

/etc/fluent/fluentd.conf
<source>
  @type tail
  path /var/log/logify/app.log
  format json
  time_format %Y-%m-%dT%H:%M:%S
  tag mylogs
</source>

<match mylogs>
  @type stdout
</match>

However, you encounter the "pattern not matched" error:

 
2024-01-27 07:13:29 +0000 [warn]: #0 pattern not matched: "{"
2024-01-27 07:13:29 +0000 [warn]: #0 pattern not matched: "      \"timestamp\": \"2024-01-27T07:13:29\","
2024-01-27 07:13:29 +0000 [warn]: #0 pattern not matched: "      \"severity\": \"ERROR\","
2024-01-27 07:13:29 +0000 [warn]: #0 pattern not matched: "      \"message\": {"
2024-01-27 07:13:29 +0000 [warn]: #0 pattern not matched: "        \"user\": {"
2024-01-27 07:13:29 +0000 [warn]: #0 pattern not matched: "          \"userId\": \"456\""
2024-01-27 07:13:29 +0000 [warn]: #0 pattern not matched: "        },"
2024-01-27 07:13:29 +0000 [warn]: #0 pattern not matched: "        \"action\": \"Payment failed\""
2024-01-27 07:13:29 +0000 [warn]: #0 pattern not matched: "      }"
2024-01-27 07:13:29 +0000 [warn]: #0 pattern not matched: "    }"

This error often has nothing to do with Fluentd, but rather, it is often caused by the source that generates the JSON logs. It is essential to verify and ensure that the JSON is valid thoroughly.

In this scenario, the source generates multi-line JSON objects, which Fluentd struggles to parse because it expects single-line JSON objects. Here's an example of a problematic log entry:

 
{
  "timestamp": "2024-01-27T07:11:30",
  "severity": "ERROR",
  "message": {
    "user": {
      "userId": "456"
    },
    "action": "Payment failed"
  }
}

To resolve this, you need to modify the source that generates the logs to output single-line JSON objects, like this:

 
{"timestamp": "2024-01-27T07:29:43", "severity": "ERROR", "message": {"user": {"userId": "456"}, "action": "Payment failed"}}

With this modification, Fluentd can successfully parse the JSON log:

 
2024-01-27 07:31:13.727430887 +0000 mylogs: {"timestamp":"2024-01-27T07:31:13","severity":"ERROR","message":{"user":{"userId":"456"},"action":"Payment failed"}}

This error can sometimes be attributed to mismatches in date formats within your JSON logs. Therefore, it's essential to closely examine the format of the logs being produced when encountering this issue.

🔭 Want to centralize and monitor your logs?

Go to Logtail and start your log management in 5 minutes.

Better Uptime Dashboard

Got an article suggestion? Let us know
Explore more
How to Rotate Log Files in Fluentd?
Questions
How to Use Conditional Statements In Fluentd
Questions
Fluentd
Learn how to use Fluentd to collect, process, and ship log data at scale, and improve your observability and troubleshooting capabilities.
Guides
How to Use Environmental Variables in Fluentd Config
Questions
Licensed under CC-BY-NC-SA

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Make your mark

Join the writer's program

Are you a developer and love writing and sharing your knowledge with the world? Join our guest writing program and get paid for writing amazing technical guides. We'll get them to the right readers that will appreciate them.

Write for us
Writer of the month
Marin Bezhanov
Marin is a software engineer and architect with a broad range of experience working...
Build on top of Better Stack

Write a script, app or project on top of Better Stack and share it with the world. Make a public repository and share it with us at our email.

community@betterstack.com

or submit a pull request and help us build better products for everyone.

See the full list of amazing projects on github