Creating Grafana Dashboards for Prometheus: A Beginner's Guide

Prometheus and Grafana are two powerful open-source tools that, when combined, form a robust solution for monitoring and visualizing the health of your systems.

This tutorial provides a comprehensive guide to setting up Prometheus and Grafana, integrating Node Exporter for metric collection, and crafting insightful dashboards tailored to your specific needs.

By the end of this tutorial, you'll have a fully functional monitoring stack, complete with dynamic dashboards that adapt to your environment.

You'll also learn how to simplify queries with variables and leverage pre-built dashboards to save time and enhance your monitoring capabilities.

Let's get started!

Prerequisites

To follow through with this tutorial, you only need a recent version of Docker installed on your machine. It'll also be helpful to know the basics of Prometheus monitoring.

Setting up Prometheus and Node Exporter

Before you can start creating dashboards with Grafana, you'll need a source of metrics data. In this section, we'll set up Prometheus and Node Exporter to collect system-level metrics on Unix systems which will later be visualized in Grafana.

The easiest way to get started is by running both services using Docker Compose, which simplifies running multi-container services on a single machine.

Here's the Docker Compose file in full:

[compose.yaml]
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    ports:
      - 9090:9090

  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - --path.procfs=/host/proc
      - --path.rootfs=/rootfs
      - --path.sysfs=/host/sys
      - --collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)
    ports:
      - 9100:9100

networks:
  default:
    name: prometheus-grafana
    driver: bridge

volumes:
  prometheus_data:

This file sets up the prometheus and node-exporter services to run on ports 9090 and 9100, respectively. The prometheus service requires a configuration file, which you must place in the same directory as the Compose file:

global:
  scrape_interval: 10s

scrape_configs:
  - job_name: node-exporter
    static_configs:
      - targets:
          - 'node-exporter:9100'

This configuration file instructs Prometheus to scrape metrics from the node-exporter service running at port 9100 on the same Docker network.

The node-exporter is already configured to collect host metrics by binding to the /proc, /sys, and / directories. This setup allows it to gather detailed information about the host's system resources, including CPU usage, memory consumption, and disk I/O, and expose them in the Prometheus' metrics format.

To see the services in action, launch them by running the following command in the Compose file's directory:

docker compose up -d

Docker will download the prometheus and node-exporter images to your machine if they are not already present, and the containers will start in the background:

. . .
[+] Running 3/3
 ✔ Network prometheus-grafana  Created                     0.3s
 ✔ Container node-exporter     Started                     0.8s
 ✔ Container prometheus        Started                     0.7s

You can confirm their status by running:

docker compose ps

The output should indicate that both services are running:

NAME            IMAGE                       COMMAND                  SERVICE         CREATED              STATUS              PORTS
node-exporter   prom/node-exporter:latest   "/bin/node_exporter …"   node-exporter   About a minute ago   Up About a minute   0.0.0.0:9100->9100/tcp, :::9100->9100/tcp
prometheus      prom/prometheus:latest      "/bin/prometheus --c…"   prometheus      About a minute ago   Up About a minute   0.0.0.0:9090->9090/tcp, :::9090->9090/tcp

To confirm that Prometheus is able to scrape the Node Exporter metrics successfully, visit http://localhost:9090/targets in your browser. You should see the node-exporter source reported as Up:

If you'd like to learn more about the metrics collected and exposed by Node Exporter, and how to customize or query them, read our comprehensive guide on the subject.

In the next section, you'll set up Grafana to visualize the metrics collected by Node Exporter.

Setting up Grafana

Grafana provides two editions of its Docker images: Grafana Enterprise and Grafana Open Source. In this tutorial, we will use the Open Source edition.

To add Grafana to your setup, modify your compose.yaml file as follows:

[compose.yaml]
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    ports:
      - 9090:9090

  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - --path.procfs=/host/proc
      - --path.rootfs=/rootfs
      - --path.sysfs=/host/sys
      - --collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)
    ports:
      - 9100:9100

  grafana:
    image: grafana/grafana-oss:latest
    container_name: grafana
    ports:
      - 3000:3000
    volumes:
      - grafana_data:/var/lib/grafana
    depends_on:
      - prometheus

networks:
  default:
    name: prometheus-grafana
    driver: bridge

volumes:
  prometheus_data:
  grafana_data:

The Grafana instance is set up to listen on http://localhost:3000, its default port. To add Grafana to your running Docker Compose stack, execute the following command:

docker compose up -d

This command will create and start the new grafana service without interrupting the already running services. You should see output similar to the following:

[+] Running 3/3
 ✔ Container node-exporter  Running                        0.0s
 ✔ Container prometheus     Running                        0.0s
 ✔ Container grafana        Started                        0.5s

To confirm that the grafana service is running, use:

docker compose ps grafana

You should see output like this:

NAME      IMAGE                    COMMAND     SERVICE   CREATED         STATUS         PORTS
grafana   grafana/grafana:latest   "/run.sh"   grafana   8 minutes ago   Up 8 minutes   0.0.0.0:3000->3000/tcp, :::3000->3000/tcp

You can now visit http://localhost:3000 to access the Grafana user interface. You will be greeted by the login page:

The default username is admin, and the default password is also admin. Upon logging in, you will be prompted to change the default password. You can choose to do so immediately or skip this step for now:

Once logged in, you will be taken to the Grafana homepage:

In the next section, you will configure your running Prometheus server as a data source in Grafana.

Configuring the Prometheus data source

Before you can start using Grafana for dashboarding, you need to connect at least one data source.

To get started, navigate to Connections > Add new connection in the Grafana menu or go to http://localhost:3000/connections/add-new-connection:

Use the search bar to locate the Prometheus option under Data Sources, and click on it:

Click the Add new data source button on the next page:

In the configuration form, find the Connection section, and enter http://prometheus:9090 as the Prometheus server URL:

You can leave the other fields as is, then scroll to the bottom and click Save & test:

You will see the following confirmation message:

With your Prometheus data source connected, you are now ready to create your first Grafana dashboard. This will be covered in the next section.

Creating a new Grafana dashboard

To create a new dashboard, navigate to the Dashboards section in the menu, or go to http://localhost:3000/dashboards. Click the New > New dashboard button on the resulting page:

Grafana provides several options at this stage. You can:

• Add a new visualization from scratch.
• Import a library of shared visualizations.
• Import an entire dashboard from a file or other Grafana users.

In this section, I'll demonstrate how to create a dashboard from scratch. Later in the guide, we'll explore importing pre-built dashboards as well.

Click the Add visualization button, then select the previously added prometheus data source:

This will open the edit view for a new panel. Here, you'll find:

• The Visualization section, where the resulting chart or graph will appear.
• The Query section, where you'll input the queries to fetch data.
• The Panel options section, where you can customize the panel's appearance.

For our first visualization, let's display the uptime of the monitored machine. Use the following PromQL query:

node_time_seconds{instance="node-exporter:9100",job="node-exporter"} - node_boot_time_seconds{instance="node-exporter:9100",job="node-exporter"}

The node_time_seconds metric provides the current time in seconds while node_boot_time_seconds metric provides the boot time of the node in seconds. Subtracting node_boot_time_seconds from node_time_seconds calculates the uptime of the node in seconds.

In the Query tab, switch to the Code mode and enter the above query. Then click Run queries:

By default, Grafana uses a time series graph to display data points against a time axis. Since we intend to display the node uptime value in human-readable form, change the visualization to Stat:

You'll see something like this:

The Stat visualization will display a large, prominent number, ideal for quickly conveying key information.

Currently, the raw value is displayed in seconds, which isn't very helpful. To make it more readable, set the Unit field to seconds (s) under Standard options. This converts the uptime into a human-readable format:

To simplify the display, scroll to Stat styles, change Color mode and Graph mode to None. This removes unnecessary visual elements, leaving a clean uptime display:

If you're satisfied with the result, update the Title and Description under Panel options, and click Apply to add it to your dashboard:

Finally, save your dashboard by clicking the save icon:

Give the dashboard a name and click Save:

And there you have it! Your dashboard is successfully updated with its first panel.

In the next section, we'll explore creating a time series graph.

Creating a time series chart

To enhance your dashboard, let's add a new panel that visualizes the memory usage of the monitored node over time. Here are some relevant metrics for memory usage:

• node_memory_MemTotal_bytes: Total physical memory available on the system.
• node_memory_MemFree_bytes: Amount of physical memory that is unused and available.
• node_memory_Buffers_bytes: Memory used by kernel buffers.
• node_memory_Cached_bytes: Memory used for file system caches. This memory is typically available for applications if needed.
• node_memory_SwapTotal_bytes: Total swap memory available.
• node_memory_SwapFree_bytes: Free swap memory available.

In this section, you'll create a time series chart showing the total memory and memory used over time.

Start by clicking the Add > Visualization button to create a new panel in your existing dashboard:

To display the total amount of RAM available, enter the following query into the Metrics browser input under the A query panel, then click Run queries:

node_memory_MemTotal_bytes{instance="node-exporter:9100",job="node-exporter"}

You will see a straight line since the total memory capacity of the node doesn't change over time:

To make things more interesting, add another query by clicking the Add query button at the bottom of the Query tab. Enter the following PromQL query to calculate memory usage, then click Run queries:

node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes

You will see the following result:

The yellow line represents memory usage, which you can now compare to the total memory available.

To make the graph easier to read, update the Legend for each query to Total Ram and Used Ram respectively, then set the Unit to bytes(IEC) as highlighted below:

Your graph is now more intuitive and easier to understand. Add a title and description to the panel as described earlier, then click Apply to save it to your dashboard:

Don't forget to save your dashboard after adding the new panel to ensure your changes are preserved.

Creating a Gauge chart

A gauge chart is ideal for displaying values that represent a single point in time, such as utilization, capacity, or thresholds. The Node Exporter exposes several metrics that are well-suited for visualization using gauge charts, including CPU usage, memory consumption, disk utilization, and network bandwidth.

For demonstration purposes, let's visualize the amount of disk space remaining on the node as a percentage. You can do this by adding a new visualization and selecting the Gauge chart:

Use the following PromQL query to calculate the filesystem usage percentage for the root mount point:

1 -  (node_filesystem_avail_bytes{mountpoint="/",instance="node-exporter:9100"} / node_filesystem_size_bytes{mountpoint="/",instance="node-exporter:9100"})

Enter the query into the query input field and click Run queries. The raw value, which will be between 0 and 1, will be displayed:

When plotting a Gauge chart for values with a known range, you can enter them in the Min and Max fields respectively under Standard options. You can also specify the Unit as Percent (0.0-0.1) as shown below to display the value as a percentage:

You can also add color-coding to the different value thresholds so that you can quickly see what nodes are dangerously close to using up all their disk space.

Scroll down to the Thresholds section, and change the Thresholds mode to Percentage. By default, values above 80% is considered high and displayed in a bright red colour:

You can adjust this depending on your needs. For example, you can add a "warning" threshold that is between 80-90% utilization and a "danger" threshold for 90% and above:

Once satisfied with the settings, give the panel a title and description, then click Add to add it to your dashboard.

With the gauge chart added, you're now ready to simplify your PromQL queries using Grafana dashboard variables. Afterward, we'll explore how to find and import pre-existing dashboards in Grafana.

Simplifying queries in Grafana with variables

Hardcoding labels in PromQL queries can make your dashboards inflexible. If these values change or you add more nodes to monitor, your dashboard will break, and you'll need to manually update each panel. To avoid this, you can use Dashboard variables, which make dashboards reusable, dynamic, and shareable.

To get started with variables, click the cog icon on the dashboard, then go to the Variables tab and click Add variable:

Select Query as the variable type. Query variables fetch values dynamically from your data source:

Let's add a job variable that references the job label in Node Exporter metrics. Set the Name and Label to job and Job respectively, then select Label values under Query type:

In the Label field, enter job, and for the Metric field, use node_uname_info. This ensures that the job label values are derived from the node_uname_info metric and you can use any metric that includes the job label.

Scroll down to see a Preview of the values. In this example, the value will be node-exporter. Click Apply to save the variable:

In the Settings page, click Save dashboard:

The job variable will now appear at the top of the dashboard, and its value will default to node-exporter:

To update a panel to use the variable, hover over the panel and click Edit:

Replace hardcoded job values in PromQL queries with $job. For example, update the Uptime panel query to:

node_time_seconds{instance="node-exporter:9100",job="$job"} - node_boot_time_seconds{instance="node-exporter:9100",job="$job"}

Click Run queries to verify that the panel works as expected, then click Apply and save the dashboard:

Everything should keep working the same way as before:

You can repeat this process for the instance label matcher. This allows you to dynamically monitor new nodes without manually changing label values in every visualization.

Importing dashboards in Grafana

When monitoring commonly used software or systems, you can often save significant time by importing pre-built dashboards into Grafana.

These dashboards are created by community members or the Grafana team and provide an excellent starting point for visualizing your metrics.

For instance, this Node Exporter dashboard visualizes nearly all default metrics from the Node Exporter. Here's how to import it into your Grafana instance:

Navigate to the Dashboards page at http://localhost:3000/dashboards, and click the New > New dashboard option:

On the resulting page, click the Import dashboard button:

Enter the URL of the Node Exporter Full dashboard into the input field and click Load:

https://grafana.com/grafana/dashboards/1860-node-exporter-full/

On the next page, select your Prometheus source and click Import:

The Node Exporter dashboard will now appear, and it should look like this:

You can explore the various visualizations within the imported dashboard. Customize panels to fit your specific needs or delete unnecessary ones.

Pre-built dashboards are highly flexible and can save you significant time while still allowing you to tailor them to your requirements.

Final thoughts

We've covered a lot of ground in this tutorial, and hopefully you're now well acquainted with the basics of visualizing Prometheus metrics with Grafana.

You've learned how to set up Prometheus and Grafana, create basic visualizations, and even import pre-built dashboards to accelerate your monitoring efforts.

But the journey doesn't stop here! There's a lot more to learn when it comes to monitoring with Prometheus and Grafana so ensure to check out their documentation and our other guides to learn more.

If managing your own Prometheus and Grafana infrastructure feels like a burden, consider a fully managed observability platform like Better Stack.

Better Stack provides a streamlined solution for collecting, visualizing, and analyzing your Prometheus metrics, along with other valuable observability data like logs in one place.

You can easily integrate your existing Prometheus setup, and also use our pre-built dashboards to get started quickly with visualization. Sign up for a free account here.

Thanks for reading, and happy monitoring!

Article by

Ayooluwa Isaiah

Ayo is a technical content manager at Better Stack. His passion is simplifying and communicating complex technical ideas effectively. His work was featured on several esteemed publications including LWN.net, Digital Ocean, and CSS-Tricks. When he's not writing or coding, he loves to travel, bike, and play tennis.

Got an article suggestion? Let us know

Next article

Relabeling in Prometheus: A Complete Guide

Learn all about relabeling in Prometheus with detailed explanations and examples of practical use cases

→

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.