How to Parse Nested JSON Fields in Fluentd
Here is how you can parse log entries with nested JSON fields. Consider the following log entry:
{"timestamp":"2024-01-27T08:33:17","severity":"ERROR","message": "{\"user\":{\"userId\":\"456\"},\"action\":\"Payment\\ failed\"}"}
If you have a Fluentd configuration like this:
<source>
@type dummy
dummy '{"timestamp":"2024-01-27T08:33:17","severity":"ERROR","message": "{\"user\":{\"userId\":\"456\"},\"action\":\"Payment\\ failed\"}"}'
format json
tag mylogs
</source>
<match mylogs>
@type stdout
</match>
You'll notice that Fluentd does not parse the nested contents under the message field, as seen in these log entries:
2024-01-27 09:03:18.656280395 +0000 mylogs: {"timestamp":"2024-01-27T08:33:17","severity":"ERROR","message":"{\"user\":{\"userId\":\"456\"},\"action\":\"Payment\\ failed\"}"}
To resolve this, you need to use a filter with a parser. Here's how it's done:
<source>
@type dummy
dummy '{"timestamp":"2024-01-27T08:33:17","severity":"ERROR","message": "{\"user\":{\"userId\":\"456\"},\"action\":\"Payment\\ failed\"}"}'
format json
tag mylogs
</source>
<filter mylogs>
@type parser
key_name message
reserve_data true
hash_value_field message
<parse>
@type json
</parse>
</filter>
<match mylogs>
@type stdout
</match>
In this configuration, you introduce the filter block with @type parser. The key_name specifies the field (message) to be parsed. Next, the reserve_data field maintains the original data structure. After that, the hash_value_field determines where the parsed values are stored. In this case, it overwrites the original message field. Alternatively, you can use a different field name like parsed_message.
After making these changes, Fluentd will successfully parse the nested JSON content:
2024-01-27 10:03:05.016725559 +0000 mylogs: {"timestamp":"2024-01-27T08:33:17","severity":"ERROR","message":{"user":{"userId":"456"},"action":"Payment failed"}}
If you are still learning to use Fluentd, check out our guide on how to collect, process, and ship log data with Fluentd, which is a great starting point.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
-
How to Use Conditional Statements In Fluentd
Conditional statements can be seamlessly integrated into Fluentd's record_transformer plugin. Here’s how you can do it: <filter mylogs> @type recordtransformer enableruby true <record> is...
Questions -
How to Add Tags to My Fluentd Events
Here is how you can add tags to Fluentd events. Let's assume you configured Fluentd to process Nginx access logs from the access.log file with a source configuration like this: <source> @type ta...
Questions -
Fluentd
Learn how to use Fluentd to collect, process, and ship log data at scale, and improve your observability and troubleshooting capabilities.
Guides -
How to Parse Nested JSON Fields in Fluentd
Learn how to configure Fluentd for nested JSON parsing in log messages for enhanced structured logging
Questions
Make your mark
Join the writer's program
Are you a developer and love writing and sharing your knowledge with the world? Join our guest writing program and get paid for writing amazing technical guides. We'll get them to the right readers that will appreciate them.
Write for us
Build on top of Better Stack
Write a script, app or project on top of Better Stack and share it with the world. Make a public repository and share it with us at our email.
community@betterstack.comor submit a pull request and help us build better products for everyone.
See the full list of amazing projects on github