Introduction to Logfmt

In recent years, structured logging has established itself as the predominant method for recording and organizing log data, a trend underscored by the widespread adoption of JSON logging.

While JSON remains the most popular choice due to its versatility and compatibility, Logfmt has emerged as a strong contender due to its simplicity and easier readability for humans.

This article delves into the nuances of Logfmt, examining its structure and syntax, and offering a comparative analysis with other structured formats, particularly JSON.

Let's get started!

What is Logfmt?

Logfmt is a logging format that first emerged at Heroku. It is notable for its compact structure of key=value pairs and its popularity stems from being both easily readable for humans and effortlessly parseable by machines.

In this format, each key/value pair is distinctly separated by a space, with an equal sign (=) marking the division between a key and its associated value:

level=INFO host=Ubuntu msg="Connected to PostgreSQL database"

Take the level=INFO pair in the example above: level is the key, and INFO is its value, indicating the log entry's severity level.

Handling nested data in Logfmt

To accommodate nested data within Logfmt, a dot notation can be utilized. This approach simulates nested structures within the flat format of Logfmt. An example of this notation is shown below:

level=INFO host=Ubuntu msg="Application running smoothly" properties.status=OK
properties.details="No errors detected" properties.performance=Optimal

In this example, fields such as status, details, and performance are conceptually nested under the properties prefix. This technique implies a hierarchical relationship similar to nested objects in JSON:

{
 "info": "INFO",
 "msg": "Application running smoothly",
  "properties": {
    "status": "OK",
    "details": "No errors detected",
    "performance": "Optimal"
  }
}

It's important to note that most Logfmt implementations treat each key/value pair independently, without recognizing any nested structure suggested by the dot notation.

Take, for instance, the behavior of the well-known Logfmt JavaScript library when parsing our earlier example:

import logfmt from 'logfmt';

console.log(
  logfmt.parse(
    'level=INFO host=Ubuntu msg="Application running smoothly" properties.status=OK properties.details="No errors detected" properties.performance=Optimal'
  )
);

{
  level: 'INFO',
  host: 'Ubuntu',
  msg: 'Application running smoothly',
  'properties.status': 'OK',
  'properties.details': 'No errors detected',
  'properties.performance': 'Optimal'
}

This result demonstrates that while the dot notation can help in organizing and conceptually grouping related data, Logfmt inherently treats such structures as flat.

Therefore, Logfmt should not be used if your logging requirements heavily lean towards representing and manipulating hierarchical data.

Logfmt best practices

When implementing Logfmt in your applications, follow these guidelines for optimal results:

• To avoid parsing ambiguity, don't use spaces in keys and opt for recognized naming conventions like Snake Case or Camel Case.

• Your keys should only contain alphanumeric characters.

• Avoid using nested properties in Logfmt as it is designed to be a flat structure format.

• Terminate distinct log lines with a newline character (\n). Your logging framework should do this automatically.

• In development environments, colorize the keys to make the log lines easy to scan for humans.

Logfmt vs JSON

JSON is currently the most widely-used structured format for logs, offering a high degree of flexibility and compatibility with various logging tools and systems.

It is natively supported by most programming languages, and many logging tools are designed to effortlessly ingest and process JSON log data out-of-the-box. It also allows for nested data structures, making it ideal for complex or hierarchical data.

However, JSON's verbosity can be a drawback compared to the more concise Logfmt. The extensive use of braces, quotation marks around keys, and commas for key-value pair separation results in visual clutter making JSON logs less readable by humans, especially when dealing with large volumes of data.

Despite these challenges, its comprehensive structure and wide tooling support often outweigh its verbosity, making it the preferred choice to satisfy most production logging requirements.

To get the best of both worlds, consider configuring your logging framework to produce Logfmt output in development and JSON in production. Here's an example using Go's Slog package:

func main() {
    opts := &slog.HandlerOptions{
        Level: slog.LevelDebug,
    }

    var appEnv = os.Getenv("GO_ENV")

    var handler slog.Handler = slog.NewTextHandler(os.Stdout, opts)
    if appEnv == "production" {
        handler = slog.NewJSONHandler(os.Stdout, opts)
    }

    logger := slog.New(handler)

    logger.Warn(
        "storage is 90% full",
        slog.String("available_space", "900.1 MB"),
    )
}

In development environments, you'll observe the following output:

time=2023-11-23T13:00:04.365+02:00 level=WARN msg="storage is 90% full" available_space="900.1 MB"

When GO_ENV is set to production, then a JSON output will be produced instead:

{
  "time": "2023-11-23T13:01:12.745540888+02:00",
  "level": "WARN",
  "msg": "storage is 90% full",
  "available_space": "900.1 MB"
}

Logfmt implementations

A variety of modules or libraries are available in most programming languages to facilitate reading, parsing, and generating Logfmt entries. Some prominent examples are:

• Node.js
• Java
• Python
• Go
• Elixir
• Ruby
• Rust
• Clojure
• Erlang
• Haskell
• Dart
• PHP
• Swift
• .NET

Final thoughts

In this piece, we've explored the Logfmt format, delving into its strengths and drawbacks when compared to JSON, the leading structured logging format.

To begin implementing Logfmt in your application, refer to the library resources mentioned earlier.

For further exploration, consider browsing our extensive logging guides to learn more techniques for effective logging in production.

Thanks for reading, and happy logging!

Article by

Stanley Ulili

Stanley is a freelance web developer and researcher from Malawi. He loves learning new things and writing about them to understand and solidify concepts. He hopes that by sharing his experience, others can learn something from them too!

Got an article suggestion? Let us know

Next article

Introduction to JSON Logging: Techniques and Best Practices

Explore the best practices for JSON logging, including structuring, centralization, and enhancing logs for efficient analysis and management

→

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.