How to Visualize Prometheus Metrics with Grafana

Prometheus is a monitoring solution that allows you to collect metrics from various sources including physical and virtual servers, individual applications, various databases, containers, Internet of Things (IoT) devices, and others. It includes hundreds of ready-made solutions for collecting data and monitoring a wide variety of services simultaneously.

However, when it comes to visualizing the collected data, the built-in options provided by Prometheus are limited. Therefore, Grafana is commonly used as a tool for visualizing the data collected and stored in Prometheus. It is a leading open-source tool for visualizing time-series data, and it lets you create graphs and dashboards based on data from more than 100 sources such as Prometheus, Graphite, InfluxDB, Zabbix, and many others.

In this tutorial, you'll install Grafana on a Linux server and configure it to plot monitoring data from a Prometheus source. By following through, you will become familiar with the following aspects of using Grafana to query and monitor Prometheus data.

Prerequisites

To complete this tutorial, you will need access to two servers running any Linux distribution, such as Ubuntu. This could be your local machines, virtual machines, or remote servers. On the first server, you should install Prometheus for data collection, and on the second, you should install Node Exporter. You can use our previous tutorial to set up these two servers.

If you don't have access to two servers, feel free to install Prometheus and Node Exporter on a single server. Optionally, you also need a registered domain name and an A record pointing to the public IP address of the server where Prometheus is installed.

Step 1 — Installing Grafana

In this step, we will install and run Grafana on the first server (the one on which Prometheus is installed on). Go ahead and log in to the server using the command below:

ssh <your_first_server_ip>

Before proceeding with the installation, there are a few additional packages that need to be installed:

sudo apt install apt-transport-https wget

The first one (apt-transport-https) allows the use of repositories accessed via HTTPS, while the second one (wget) is a tool for downloading files over the network. Now, you can download the Grafana GPG key and add it to the list of trusted keys:

wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

Afterward, add the Grafana repository for stable releases:

echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

Next, refresh your APT cache:

sudo apt update

You are now ready to install Grafana on your Ubuntu server:

sudo apt install grafana

After the package is downloaded and installed, you can launch the Grafana server with systemctl:

sudo systemctl start grafana-server

Confirm that it started correctly through the command below:

sudo systemctl status grafana-server

You should see some output similar to the following:

● grafana-server.service - Grafana instance
     Loaded: loaded (/lib/systemd/system/grafana-server.service; disabled; vendor preset: enabled)
     Active: active (running) since Tue 2021-11-23 15:05:41 UTC; 7s ago
       Docs: http://docs.grafana.org
   Main PID: 16376 (grafana-server)
      Tasks: 8 (limit: 4415)
     Memory: 27.9M
     CGroup: /system.slice/grafana-server.service
. . .

If you have a firewall configured and enabled, you must allow traffic to port 3000:

sudo ufw allow 3000

Finally, enable the service to run Grafana automatically on boot:

sudo systemctl enable grafana-server.service

Synchronizing state of grafana-server.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable grafana-server Created symlink /etc/systemd/system/multi-user.target.wants/grafana-server.service → /lib/systemd/system/grafana-server.service.

Now you have a Grafana server running with basic settings, which is enough for the first use. Before we proceed with configuring Grafana, we need to ensure that the connections to and from the server are secured over HTTPS. This is what we will try to achieve in the next section, but feel free to skip it if you don't have a registered domain name for your server.

Step 2 — Setting up HTTPS for Grafana

To ensure that no one intercepts our data (for example, logins and passwords), you need to set up a secure connection. For this, you need an SSL certificate. You can generate and sign one yourself, but browsers will constantly show a warning due to a lack of trust.

In this section, we will show how to get a valid and recognized certificate, and use it with Grafana provided that you have a registered domain name pointed to your server's IP address. Let's Encrypt is a free, automated, and open certification center that will help us achieve this.

The first step here is to install the certbot package. It's a utility that allows you to generate and renew Let's Encrypt certificates automatically:

sudo apt install certbot

If you have a firewall configured and enabled, you need to allow traffic on ports 80 and 443. The former is required by certbot to obtain a certificate, and the latter is the standard port for HTTPS traffic.

sudo ufw allow 80

sudo ufw allow 443

You can now run certbot to obtain a Let's Encrypt certificate. Ensure that no other service is running on port 80 before running the command below. For example, you may need to stop the nginx service first if you followed our previous tutorial on Prometheus:

sudo systemctl stop nginx

Ensure to replace the domain name in the example below with your own before executing the following command:

sudo certbot certonly --standalone -d yourdomain.com

When running the command, you will be prompted to enter an email address and agree to the terms of service. If the required ports are open and there are no errors, you will observe the following output:

Obtaining a new certificate
Performing the following challenges:
http-01 challenge for yourdomain.com
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/yourdomain.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/yourdomain.com/privkey.pem
   Your cert will expire on 2022-03-14. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Since Grafana is launched as an unprivileged user, you should copy the certificates to its directory and change the ownership of the files:

sudo cp /etc/letsencrypt/live/yourdomain.com/fullchain.pem /etc/grafana/fullchain.pem

sudo cp /etc/letsencrypt/live/yourdomain.com/privkey.pem /etc/grafana/privkey.pem

sudo chown grafana:grafana /etc/grafana/*.pem

Next, open the Grafana configuration file:

sudo nano /etc/grafana/grafana.ini

Find the [server] section, uncomment the protocol, cert_file and cert_key options. Then change their values as shown below:

#################################### Server ####################################
[server]
# Protocol (http, https, h2, socket)
protocol = https

. . .

# https certs & key file
cert_file = /etc/grafana/fullchain.pem
cert_key = /etc/grafana/privkey.pem

Save the configuration file and restart the service for the changes to take effect:

sudo systemctl restart grafana-server

The connection to and from the Grafana is now fully secured, so we can now get on with the rest of the configuration.

Step 3 — Updating Grafana's administrative credentials

Grafana is automatically configured with a default username and password with every new installation. It is good practice to change it to improve security. This step will show you how to do this when you first log in to the Grafana web interface.

Open your web browser and navigate to https://yourdomain.com:3000 or http://<your_server_ip_address>:3000. You should observe the following welcome screen:

Enter admin into both Email or username and Password fields, then click on the Log in button. Afterward, you will be prompted to change your password:

Enter your new password twice and confirm the password change by clicking on the Submit button. You can also click the Skip button to skip this step and leave the default password (not recommended!). Once you're done, you will see the default Grafana dashboard:

At this stage, no one except yourself or anyone who knows the administrator password will be able to make changes using the Grafana web interface. In the next step, we will connect our running Prometheus server as a data source to this Grafana instance.

Step 4 — Configuring the Prometheus Data Source

Before you can start using Grafana for monitoring, you need to connect at least one data source. In this tutorial, we will use the previously configured Prometheus server as the data source.

Adding a new data source to Grafana is done by clicking on the gear icon in the left side of the screen and selecting the Data sources item. You will see a page indicating that there are no data sources yet.

Clicking the Add data source button will take you to a page listing the available sources that are shipped with Grafana by default. This listing can be expanded by installing additional plugins. The Prometheus source we need here is at the very beginning of the list, so hover the mouse over it and click the Select button.

You will see the configuration page for the Prometheus data source:

Since we installed Grafana on the same server as Prometheus, we only need to change a couple of options. Let's go through the basic settings:

• Name: setting a unique name for the data source here is useful for differentiating between multiple Prometheus servers.
• Default: turning this option on causes this data source to be preselected in new panels that you create.
• URL: this option sets the full path to the Prometheus API. In our case, it's http://localhost:9090.

You can leave all the other options as is, and scroll down the page until you find the Save & test button. When you click this button, you will see the following confirmation message:

Now that the Prometheus data source is successfully configured, let's take a look at the dashboard that ships with the plugin in the next section.

Step 5 — Importing and exploring the Prometheus stats dashboard

Several pre-configured dashboards are supplied with the Prometheus plugin for Grafana. Without leaving the data source settings, navigate to the Dashboards tab. You will see a list of dashboards available for importing:

Select any dashboard from the list and click on the Import button. Afterward, click on its name to go to the dashboard page. You will observe plots of various metrics related to the Prometheus server, such as the count of samples appended by the server, the duration for all evaluations to execute, memory consumption, Write-Ahead Log (WAL) corruptions, and others.

In the next section, we will create a custom dashboard that displays one of the metrics received from our Node Exporter server on it.

Step 6 — Creating a custom Prometheus dashboard

In this section, we will create a custom dashboard and add a graph plotting the utilization of the root partition on our Node Exporter server. To add a new dashboard, click on the + icon in the menu on the left side of the screen and select the Dashboard item. A new empty dashboard will be displayed and you will also be prompted to choose between a new row or a new panel.

A Grafana dashboard can contain the following elements:

• Panels: all information will be displayed in them. You can choose the type of visualization: regular graph, bar graph, table with data, single value. There are over a dozen different visualizations available.
• Rows: they can be used to group panels of the same type. You can find more information about Rows in the documentation.

We will create a simple dashboard with a single panel for demonstration purposes. To achieve this, select the Add a new panel option. Subsequently, you will see a new panel and its settings:

Since we have set the Prometheus source as default, it will be automatically selected. We now need to enter the metric that we want to display on the graph. To do this, enter the following in the Metrics browser field:

node_filesystem_avail_bytes{ job="node_exporter_client", mountpoint="/" }

We'v used the job name that we configured in Step 8 of our previous article on Prometheus. Ensure to click the Run query button afterward.

On the right side of the Edit Panel page, you'll see the settings that relate to the visualization. There are quite a few of them, and they will vary for each specific type of visualization.

Let's change the title for our new panel by entering the following text in the Title field. After making a change in the settings, it will reflect immediately in the panel.

Free space on /

Let's now change the units on the graph. There is a separate option for this which can be found by typing unit in the search field in the upper right corner. Select Data and then bytes(IEC) from the presented dropdown menu.

Finally, click the Save button on the top of the page to save your changes. In the dialog that opens, enter your dashboard's name and confirm the changes by clicking on the Save button.

Your custom dashboard is now saved! In the future, you can add other metrics that you want to monitor to it. In the meantime, let's discuss where you can find ready-made dashboards and how to import them to your Grafana instance in the next section.

Step 7 — Exploring and importing community dashboards

Sometimes it might make more sense to find a ready-made dashboard and customize it as needed instead of creating one from scratch. There are more than 5000 community dashboards available at the time of writing. To see a list of currently available dashboards, follow this link in your browser. You will be taken to a page containing all dashboards created by other Grafana users. On the left side of the page, there are options for filtering the list.

For this tutorial, we will look for dashboards where the source is Prometheus. To do this, select the Prometheus option from the Data source drop-down list. In a second, the page will be refreshed and you will see a filtered list of dashboards.

There are more than 2000 different dashboards available that relate to Prometheus so let's find one that you are already familiar with. Scroll down the list and find the Prometheus Stats dashboard:

Clicking the entry will redirect you to a page that contains general information about the dashboard, the minimum version of Grafana required for it to work, and what other plugins are needed. Each dashboard also has a unique ID, so make sure to copy it by clicking the Copy ID to Clipboard button.

Return to your Grafana instance, and click on the + icon on the left side of the screen. Select the Import item, then paste the copied ID into the Import via grafana.com field and click the Load button next to the input.

Afterward, you will be presented with several options:

• Name: replace the default name with one of your choice.
• Folder: choose in which folder to save the dashboard.
• Unique identifier (UID): this option is useful if you already have external links to this dashboard and you want them not to break.
• Prometheus: you need to select a pre-configured Prometheus data source from the list.

Confirm the creation of a new dashboard by clicking on the Import button. You will be immediately redirected to the new dashboard.

As you can see, you don't always need to create a custom dashboard from scratch, because you can search for a ready-made one that fits your requirements. Ensure to read the description properly before using the dashboard so you can find out if there are instructions on how to display the data and what tools are required to get it working.

Final thoughts

In this tutorial, you learned how to install Grafana and how to use it to query and monitor Prometheus data. We discussed how to used the built-in Prometheus dashboards, and how to create a custom one from scratch. We also covered how to utilize community-sourced dashboards.

To learn more about setting up Grafana for monitoring Prometheus data, refer to the relevant documentation or read our Prometheus vs. Grafana comparison.

Thanks for reading, and happy monitoring!

Article by

Vadym

Got an article suggestion? Let us know

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.