A Complete Guide to Logging in Heroku

Heroku is a cloud platform that allows developers to build, deploy, and manage their applications in the cloud without worrying about the underlying infrastructure. It is a fully managed platform, so there is no need for developers to manage and maintain servers, hardware, or operating systems. This means that you can focus on building your applications without worrying about administering server infrastructure.

Logging is a crucial aspect of application monitoring, and Heroku provides several tools to centralize the logging experience across the entire platform. These tools allow you to record events pertaining to your application's activities, and use the collected data to gain insights into the application's behavior, performance, and reliability.

In this article, we will delve into the process of logging in Heroku, and how to bypass its limitations by forwarding your logs to other log management platforms.

Prerequisites

Before proceeding with this article, ensure that you already understand how to create and deploy a Heroku application. You also need to understand basic logging concepts such as log levels, and log rotation.

Setting up the demo application

To help you better understand logging in Heroku, we have created a demo application which you can fork and deploy to Heroku.

Head to the demo app's GitHub page and create a fork.

Afterward, create a new Heroku application.

Link the forked repository to the Heroku app.

Finally, deploy the main branch to production. It will start generating logs immediately its deployed.

Heroku logging basic concepts

Heroku offers a range of logging capabilities similar to other cloud platforms. However, it also has some unique features that set it apart from the competition. For example, it provides a tool called Logplex, which acts as a log stream router, collecting logs from different sources and merging them into one single channel, making it easier for you to observe. This architecture is illustrated in the diagram below:

Sources

Let's start with the sources to understand precisely how this process works. Logplex can collect logs from the following sources:

• App logs: These are logs generated by the code and dependencies of the application itself.
• System logs: These logs contain messages about actions taken by the Heroku platform infrastructure, such as restarting a crashed process, sleeping or waking a web dyno, or serving an error page due to a problem in your app.
• API logs: These logs contain messages about administrative actions taken by you and other developers working on your app, such as deploying new code, scaling the process formation, or toggling maintenance mode.
• Add-on logs: These logs contain messages from add-on services.

There is another type of log called the build log which is generated while building and deploying your application. These logs are treated differently and can only be accessed through the Heroku dashboard.

Logplex

Logplex is a log router that collects logs from the sources mentioned above and redirects them to a single channel. It provides a centralized logging system for applications running on Heroku, allowing developers to view and analyze log data in a single place.

Logplex streams log data in real-time, making it easy to monitor and troubleshoot issues as they occur. It also provides tools for searching and filtering log data, so developers can easily find the needed information.

In addition to providing a centralized logging system, Logplex also helps to scale applications by buffering and batching log data, allowing applications to send large volumes of log data without experiencing performance issues.

Log Drains

Heroku log drains are a feature of the Heroku Logplex service that allows you to send log data from your applications to external logging and analysis tools such as Logtail, Splunk, or Datadog. This allows you to perform advanced analysis and visualization of your log data, as well as integrate with other tools and systems.

To set up a log drain, you need to specify the URL of the external logging service you want to use, and Logplex will automatically send log data to that service. Log drains can be configured for individual applications or the entire Heroku account, depending on the developer's needs.

In addition to sending log data to external tools, log drains can also be used to send log data to custom destinations, such as a file on a server or a message queue so that you can build custom log processing and analysis pipelines that fit your specific needs.

All three components, sources, Logplex, and log drains, form the basic architecture of Heroku's logging functionality.

Writing logs

Heroku captures anything written to the standard output (stdout) or standard error (stderr), so ensure to configure your logging framework of choice to always send its output to one of the two locations.

How to view Heroku logs

Now that we've discussed how Heroku collects logs, it is time to discuss how you to view the log records. Heroku allows you to view your logs using the Heroku CLI, the dashboard, third-party add-ons, or your custom log drains.

Using Heroku CLI

To use Heroku CLI to view logs, you must have Heroku CLI installed on your system. And then log into your account using the following command:

heroku login

After you've logged in, you'll have the permission to retrieve log records associated with your account. To get started, you need to remember the following commands. To get the most recent log records, use the following command:

heroku logs

However, if you run this command, an error message will be returned, that is because you must specify the exact app from which you wish to pull the log records. You can do that using the -a or --app flag:

heroku logs -a <app_name>

. . .
2022-12-28T19:08:23.612947+00:00 app[web.1]: [2022-12-28 19:08:23 +0000] [4] [INFO] Starting gunicorn 20.1.0
2022-12-28T19:08:23.613444+00:00 app[web.1]: [2022-12-28 19:08:23 +0000] [4] [INFO] Listening at: http://0.0.0.0:57271 (4)
2022-12-28T19:08:23.613516+00:00 app[web.1]: [2022-12-28 19:08:23 +0000] [4] [INFO] Using worker: sync
2022-12-28T19:08:23.618325+00:00 app[web.1]: [2022-12-28 19:08:23 +0000] [9] [INFO] Booting worker with pid: 9
2022-12-28T19:08:23.628735+00:00 app[web.1]: [2022-12-28 19:08:23 +0000] [10] [INFO] Booting worker with pid: 10
. . .

Pay closer attention to the output, and notice that all log records follow a certain format:

timestamp source[dyno]: message

• The timestamp is the date and time when this log record is generated.
• The source of the log line can be either app for log lines generated by your app's dynos (web dynos, background workers, cron), or heroku for log lines generated by Heroku's system components (HTTP router, dyno manager).
• The dyno is the name of the dyno or component that produced the log line. For example, a worker dyno may be named worker.1 and the Heroku HTTP router may be named router.
• The message is the content of the log line. If the log record is longer than 10,000 bytes, it will be split into chunks of 10,000 bytes without any extra trailing newlines, and each chunk will be submitted as a separate log line by Logplex.

By default, the logs command retrieves the latest 100 log lines. However, you can change that by adding a --num or -n option. This allows you to retrieve up to a maximum of 1500 lines.

heroku logs --num 200

To view Heroku log records in real-time, you can use the --tail or -t option:

heroku logs --tail

To close the real-time tail, press Ctrl+C to return to the terminal prompt.

As mentioned earlier, Heroku collects log records from many different sources. Instead of view logs from all sources at once, you can filter the output from the logs command so that only logs from specific sources or dynos are displayed. The --source and --dyno options are provided for this purpose respectively.

heroku logs --dyno router

heroku logs --source app

You may also combine the two options:

heroku logs --source app --dyno router

Using the Heroku dashboard

An alternative way to view your Heroku logs is by using the Heroku dashboard web interface. Navigate to your app and click More on the top right corner to see a dropdown menu. Select View logs to see all log records associated with this app.

Using third-party add-ons

Lastly, you may also opt to connect your Heroku application to a third-party log management service, such as Logtail.

Logtail is a log management tool used to collect log data from multiple sources and send it to a centralized cloud platform for further processing and analysis. It can collect log data from various sources, including databases, servers, and APIs. In addition, Logtail can handle log data in multiple formats, including text, JSON, and binary, making it suitable for use with a wide range of log sources.

Alternatively, if you prefer to manage the logs locally, you can set up a custom log drain to receive and view log records. Later, we will demonstrate how to set up Logtail and log drains using a real-life example.

Best practices for logging in Heroku

Heroku is a cloud platform that supports multiple programming languages, and each programming language has its way of pushing application logs. We have created detailed logging guides for many of the supported languages, such as the ones listed below:

• Ruby
• Python
• Node.js
• Java
• PHP
• Go

Regardless of what languages you are using, there are a few best practices you should keep in mind to get the most out of your logs.

1. Log as much as necessary

First of all, you should always log events that describes how your application is functioning. These events include but not limited to application errors, authentication attempts, data access and modifications, and other important actions performed in your application.

2. Add contextual information

Each log record should contain enough contextual information that describes the event. For example, when a user logs in, you shouldn't just record a simple message. Rather, you should also include some information in the record such as the user ID, timestamp, user agent, and other relevant details about the event:

2022-12-17T15:02:00.003041+00:00 app[web.1]: INFO: User 12364926 logged in. {"userid":12364926}

3. Use structured logging

Using a structured logging format ensures that your log records can be automatically processed by various logging tools which will save you time when investigating an issue. JSON is go-to structured format for most people, but other options like logfmt also exist.

4. Exclude sensitive information

Take adequate care to never log business secrets, or personal data such as email addresses, passwords, or credit card information so that you don't compromise user privacy or incur regulatory fines.

5. Use the appropriate log level

Always make sure your log records have the appropriate log level so that you can easily differentiate between events that require urgent attention from those that are merely informational.

Integrating Heroku with a logging add-on

Heroku's default options for viewing and managing logs are quite limited, not to mention its paltry 1500 log history limit. If you want to persist more data and do more with your log records, you must use a logging add-on or create your own syslog drain. Let's explore the former first.

There are several logging add-ons available for Heroku, but Logtail stands out as one of the best options for log aggregation, analysis and alerting. It can ingest logs from other platforms, not just Heroku, and it also offers team collaboration features so that different teams within your organization can to use the log data to collect different insights.

You can get started with Logtail for free by heading to the Logtail add-on page, and then click Install Logtail.

You may start with the free trial plan and choose the app you just created.

After you click Submit Order Form, you will be redirected to the Resources page. At this stage, all log records in Heroku will be redirected to Logtail and you can view them through the Live Tail page.

Creating a custom log drain

If you'd rather create a custom log drain for your Heroku logs, you can do that by setting up a log drain which could be either Syslog or HTTPS.

An HTTPS drain is usually the easiest to set up. First, you need to configure your web server so that Heroku can make POST request to it. Then run the following command:

heroku drains:add https://<username>:<password>@<url> -a <app_name>

A Syslog drain, on the other hand, requires an extra step. After running the following command:

heroku drains:add syslog+tls://<url>:<port> -a <app_name>

A drain token will be generated, and you need to retrieve this token using the following command:

heroku drains -a <app_name> --json

{
  "add-on": null,
  "created_at": "2018-12-04T00:59:46Z",
  "id": "906262a4-e151-45d2-b35a-a2dc0ea9e688",
  "token": "d.f14da5dc-106b-468d-b1bd-bed0ed9fa1e7",
  "updated_at": "2018-12-04T00:59:47Z",
  "url": "syslog+tls://newrelic.syslog.nr-data.net:6515"
}

Finally, use the token to register the newly created log drain with your log analytics app. Please refer to the Heroku documentation for more details on setting up a log drain.

Conclusion

In this article, we covered the process of logging in Heroku applications. We provided an overview of the architecture of Heroku's logging system, including the various types of logs that are available. We also explored different methods for accessing log records, and shared best practices for logging in Heroku. To close out the article, we demonstrated how to get Heroku logs on other platforms through a logging add-on or a custom log drain.

Thanks for reading, and happy logging!

Article by

Eric Hu

Eric is a technical writer with a passion for writing and coding, mainly in Python and PHP. He loves transforming complex technical concepts into accessible content, solidifying understanding while sharing his own perspective. He wishes his content can be of assistance to as many people as possible.

Got an article suggestion? Let us know

Next article

A Complete Guide to Logging in Vercel

In this article, we will discuss the basics of logging in Vercel, including different types of logs, how to view them, and how to bypass Vercel's limitations by integrating your projects with external log management platforms.

→

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.