Fluentd Log Unreadable. It Is Excluded and Would Be Examined Next Time

When you see the message "Log unreadable. It is excluded and would be examined next time" in Fluentd, it indicates that Fluentd encountered an issue while reading a log file. This issue could arise due to various reasons, such as file permissions, format issues, or file rotation. Below are some common causes and how to troubleshoot them:

1. File Permission Issues

Fluentd might not have the necessary permissions to read the log file.

• Check File Permissions: Ensure that the user running Fluentd has read permissions for the log file.

   

  Copied!

  ls -l /path/to/your/log/file.log
  

  If necessary, adjust the permissions:

   

  Copied!

  chmod 644 /path/to/your/log/file.log
  

  Or change the ownership:

   

  Copied!

  chown fluentd_user:fluentd_group /path/to/your/log/file.log
  

2. Log File Format Issues

Fluentd might have trouble reading the file due to unexpected file format or corruption.

• Verify Log Format: Check that the log file content is in the expected format (e.g., plain text, JSON, etc.).
• Check for Corruption: Ensure that the log file is not corrupted. You can check this by manually opening and inspecting the file.

3. File Rotation Issues

Log files may be rotated (e.g., by logrotate), which can cause issues with Fluentd’s tail plugin.

• Configure rotate_wait: If you are using the tail input plugin, you might want to increase the rotate_wait parameter. This parameter tells Fluentd how long to wait before checking if a rotated log file has appeared.

   

  Copied!

  [INPUT]
      Name        tail
      Path        /path/to/your/log/file.log
      Parser      your_parser
      Tag         app.logs
      Rotate_Wait 30s
  
  

4. Fluentd Buffering and Parsing Issues

If Fluentd encounters issues while buffering or parsing the log files, it might skip them.

• Check Fluentd Logs: Fluentd's own logs might provide more detailed information about the issue.

   

  Copied!

  tail -f /var/log/fluentd/fluentd.log
  

• Check Parser Configuration: Ensure that the parser configuration matches the format of the log files. For example, if you're parsing JSON, make sure the JSON structure is correct and consistent.

5. Log File Encoding Issues

Fluentd might encounter issues if the log file has an unexpected encoding.

• Check Encoding: Ensure that the log file is in the expected encoding (e.g., UTF-8). You can use the file command to check the encoding:

   

  Copied!

  file -i /path/to/your/log/file.log
  

  If necessary, convert the encoding:

   

  Copied!

  iconv -f <source-encoding> -t utf-8 /path/to/your/log/file.log -o /path/to/your/converted/log/file.log
  

6. Monitor Fluentd's Retry Mechanism

Fluentd will typically retry reading files it couldn’t read previously. Ensure the issue isn’t transient by monitoring for further errors after fixing the possible causes.

Final Steps

• Restart Fluentd: After making the necessary adjustments, restart Fluentd to ensure the changes take effect:

   

  Copied!

  sudo systemctl restart fluentd
  

• Monitor Logs: Continue to monitor Fluentd's logs to ensure that the issue has been resolved and that logs are being processed as expected.

If the problem persists, you may need to provide additional log details or Fluentd configuration settings for more targeted troubleshooting.

Got an article suggestion? Let us know

Explore more

Fluentd

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.