🔮 Want a better platform to centralize your logs?
Go to Logs and start your log management for free in 5 minutes.
Logging is the process of recording events or messages that occur during the execution of a software program. It is a crucial component in the software development cycle. And Microsoft Azure, as a cloud computing platform and infrastructure created by Microsoft for building, deploying, and managing applications, provides a comprehensive logging solution for its various services, known as Azure Monitor. It allows you to collect, analyze, and visualize logs generated by your applications, services, and infrastructure.
In this tutorial, we will discuss how to start logging in Azure, including collecting, viewing, and searching log entries, visualizing log data by creating charts, and setting up a log-based monitoring system.
Go to Logs and start your log management for free in 5 minutes.
Azure Monitor is a built-in data platform in Azure that provides monitoring and diagnostics for resources and applications. It helps you understand the performance and status of your resources, detect issues, and take corresponding actions. In addition, Azure Monitor supports log and metric data collection, alerting, dashboards, and integrations with other Azure services. The following diagram from Azure documentation illustrates the architecture of the Azure Monitor service.
First, on the left side of this diagram, you can see that Azure Monitor collects data from various sources, such as applications, infrastructure, Azure platform, or other custom sources. The method of accessing data from different sources varies, as we are going to discuss later. These data will be stored inside Azure Monitor as metrics, logs, traces, or changes.
Azure Monitor will then be able to process these data for further analysis/ visualization or integrate with other services of the Azure platform. You can also export the data to third-party applications such as Better Stack Logs through import/export APIs. For this article, we will focus on how to use Azure Monitor to collect and process log data.
Before we can talk about collecting log data, you must first understand how Azure organizes its infrastructures. Azure applications are divided into different tiers. They are the Azure tenant, which is at the bottom level, and Azure Subscription, Azure Resources, operating system, as well as the application source code, which is at the top level.
Azure Monitor treat each tier as an independent data source, and they have different data collection methods.
Let us start from the bottom level, the Azure tenant. Data related to the Azure tenant can be collected from the Azure Active Directory report, which contains sign-in activities and audit logs regarding all tasks performed by this particular tenant.
At the Azure Subscription level, every Subscription will generate a trail of activity logs, and you may export these logs to a Log Analytics workspace for further analysis.
As for the Azure Resources, each Resource will generate logs providing insights into the internal working of the Resource. These logs are collected automatically, but you need to update the diagnostic setting so that they may be forwarded to Azure Monitor.
Next, for the operating system, you can collect log data automatically by installing the Azure Monitor agent. The agent can be installed using the Azure CLI, and then it will then be able to deliver monitoring data to Azure Monitor for future use.
After the agent has been installed, you should also configure how the data will be collected by defining a data collection rule and associating the operating system to the rule.
The following data can be forwarded to Azure Monitor through the agent:
At the application level, Azure has provided SDKs for some popular programming languages and technologies to make things easier for you. For now, the supported technologies include .NET, Java, JavaScript, Node.js and Python. These SDKs allow you to send log data to Azure Monitor directly. Please refer to their respective documentation for details.
In addition to the standard tiers of an application, Azure Monitor also allows you to send data from custom sources using the logs ingestion API.
To use this API, you must create a data collection endpoint for Azure Monitor. The endpoint allows you to send data directly to Log Analytics workspaces.
Next, you should define a data collection rule to specify the format the log data.
And lastly, make an API call to the endpoint in your application. Its payload should have the log data formatted in JSON. The call should point to a specific table in the workspace, follow a specific format and apply a data collection rule so that Azure Monitor can understand the log data. If the format does not match the data collection rule, The API call will need to go through a data ingestion pipeline, transforming the data for the target table.
The following image from Azure docs illustrates this process:
Now that you have successfully sent log records to Azure Monitor, let's discuss how you can view and analyze them. To do this, you must create a Log Analytics workspace. Log Analytics workspace is part of the Azure Monitor platform designed to process and analyze log data. Each workspace has its own data repository, allowing you to combine data from multiple sources.
This interface consists of three main components. First, you will find multiple tables on the left side, each corresponding to a single data source. The tables are all defined with a unique set of columns. You may use log queries to retrieve specific columns of data and feed them to other services that are provided by the Azure Monitor.
The following types of tables are available in the Log Analytics workspace:
The top section is where you can search for logs using the Kusto Query Language (KQL).
And the bottom section is where the search result will be displayed.
Once you've collected the log data, you can take it one step further and extract numeric metrics from the logs to create visualizations using the Metrics Explorer.
The Metrics Explorer provides a comprehensive view of the performance of your Azure resources. With Azure Metrics Explorer, you can view real-time and historical metrics data from various Azure services, including virtual machines, web apps, databases, and more. It is a crucial tool for gaining insights into how your resources are used and how they perform over time.
Besides creating charts, the Explorer allows you to view metrics from multiple resources in a single dashboard and compare metrics data to identify trends and patterns. You can also export the created chats to other tools, such as Power BI, for further analysis and visualization.
Azure allows you to create threshold-based or log-based alerts using its Azure Monitor Alerts service. It provides real-time notifications about critical events or conditions in your Azure resources. For example, with Azure Monitor Alerts, you can create alerts triggered by specific conditions in your Azure services by setting up alert rules, such as when a metric exceeds a specified threshold or when a log query returns a particular result.
Azure Alerts can be configured to send notifications via email, SMS, or to an Azure Event Grid topic. You can also integrate Azure Alerts with other tools, such as Microsoft Teams or Slack, to receive notifications in the way that is most convenient for you.
Besides setting up a notification channel, Azure also allows you to configure some more complex automation actions, such as restarting a server, executing a script, and so on. These actions will be executed when the alert is triggered. You can also group multiple actions to form an action group, which will help simplify the process of managing your alert actions, allowing you to respond quickly to issues and resolve them before they impact your customers.
By using Azure Alerts, you can stay informed about critical events or conditions in your Azure resources and take timely action to resolve any issues.
As you may have noticed, Azure comes with a rather complex log management and monitoring system. If you wish to manage your logs elsewhere using a third-party log management service such as Logs, Azure allows you to export log data to external destinations through different channels, such as REST API or an export job.
By using REST API, Azure allows you to export log data stored in Azure Monitor Logs by setting up an API endpoint, which allows third-party applications and services to retrieve information. Or you may create a Azure Log Analytics export job, which allows you to export the log data to other Azure services include Azure Blob Storage, Event Hubs, and Power BI.
Azure Blob Storage is a cloud-based object storage service for storing unstructured data. It is highly scalable, durable, and cost-effective, making it the perfect tool for backing up and archiving log data in the long term, as it does not have a fixed log retention period.
Azure Event Hubs is a cloud-based event-processing tool that provides a highly scalable, durable event ingestion and delivery service. This option allows you to stream log data to other applications and services for further analysis.
Lastly, Power BI is a business intelligence and data visualization platform that allows you to analyze and visualize data from a wide range of sources
Lastly, there are a few things you should note when logging in Azure:
As we've discussed before, Azure Monitor is a very powerful infrastructure monitoring system. However, it is a bit complex to set up and use. If you are looking for a modern alternative to Azure Monitor that is easier to set up, consider sending your application logs to Logs instead. Better Stack Logs is a cloud-based log management solution that allows you to view, search, and process all your logs in one place.
Better Stack Logs offers many client libraries that allow you to send your application logs to Better Stack Logs directly. We have also created detailed logging guides for many different languages, such as the ones listed below:
In this article, we discussed some fundamental concepts regarding logging in Azure. Then, we explored how to collect log data from different sources, query the collected log entries, visualize them by creating charts, set up log-based alerts, and export log data to external destinations. And lastly, we listed some best practice guidelines you should follow when logging in Azure, which will make sure you are using Azure Monitor to its full potential. We hope this tutorial has helped you understand the various logging and monitoring features provided by Azure.
If you wish to dig deeper into the subject of logging, we also provide several tutorials regarding log rotation, log levels, and centralizing logs to help you build a more effective logging system.
Thanks for reading, and happy logging!