Monitoring the state of your servers and the processes running on them is
crucial to optimize settings, correct errors, or respond to increased load in
time before your users notice a degraded performance. This requires a monitoring
system that allows you to collect metrics from servers, network devices,
applications, or containers. While there are several monitoring solutions in
this space, one of the popular and multifunctional systems is
Prometheus.
Prometheus is an open source monitoring solution that consists of several
components written in the Go programming language. Its main component is a
time-series database for storing metrics, but there are other components such as
the exporters for
collecting data from various services, and the
alertmanager which
handles alerts.
This guide will provide information on how to install and configure Prometheus
and Node Exporter on your Linux
servers. The Node Exporter is a tool that exposes a wide variety of hardware and
kernel-related metrics for
Unix-like systems. There is also the
Windows exporter for
Windows that serves an analogous purpose.
By following through with this tutorial, you will become familiar with the following aspects of using Prometheus and Node Exporter to monitor Linux servers:
Installing and configuring Prometheus.
Installing and configuring Node Exporter.
Scraping Linux host data with Node Exporter.
Using an SSL certificate to secure the connection between Prometheus and
Node Exporter instance.
Preventing unauthorized access to Prometheus data by securing it behind
a proxy.
Prerequisites
To complete this tutorial, you'll need access to two Linux systems. This could
be local machines, virtual machines, or remote servers. We will use the first
system to run the Prometheus instance, which will monitor the performance of the
second system that is running a Node Exporter to scrape metrics from the OS.
If you don't have two systems available to you, then feel free to run both the
Prometheus and Node Exporter instance on the same server and adjust the commands
and instructions below as needed. To keep things simple, we'll be using Ubuntu
20.04 for both servers but the process should be near identical for other
Linux-based operating systems.
Step 1 — Preparing the servers
Before proceeding with the installation of Prometheus and Node Exporter, it is
necessary to prepare your servers by creating users and directories, opening
ports in the firewall, and generating a certificate that will secure the
connection between the Prometheus server and the Node Exporter server.
Let's begin with the first system. Log in to the server and create a user named
prometheus through the commands below:
Here, the options --no-create-home and --shell /bin/false will ensure that
no home directory is created for this user, and this user cannot login to the
server.
Next, create the following directories for storing Prometheus' configuration
files and data:
sudo mkdir /etc/prometheus
Copied!
sudo mkdir /var/lib/prometheus
Copied!
Then, change the user and group ownership on the new directories to the
prometheus user.
The following two commands are optional. If you plan to run Prometheus on the
local network and do not use a firewall, you can skip them.
Open the default HTTP port for the Prometheus server:
sudo ufw allow 9090/tcp
Copied!
sudo ufw reload
Copied!
At this point, we are done with the initial preparation of the Prometheus
server. Let's go ahead and prepare the second system, which will run a Node
Exporter instance. Login to the server and create a new user called nodeuser:
Next, create a directory for storing the Node Exporter's configuration files:
sudo mkdir /etc/node_exporter
Copied!
Node Exporter uses port 9100 for incoming connections, so the Prometheus
server will connect to this port. If you use a firewall, run the commands below
on the Node Exporter server to allow connections on this port:
sudo ufw allow 9100/tcp
Copied!
sudo ufw reload
Copied!
Step 2 — Securing the connection between the servers
We need to create an SSL certificate to secure the connections between the
Prometheus server and the Node Exporter. In this tutorial, we will be using
self-signed credentials. Run the following command on the Node Exporter server
to create a certificate:
Don't forget to replace <your_second_server_ip> string with the IP address of
your second server, to which the Prometheus server will connect. You will see
the following output:
Output
Generating a RSA private key
...+++++
...............................................+++++
writing new private key to 'node_exporter.key'
-----
As a result, you will get two files in the current directory:
node_exporter.crt and node_exporter.key. Copy these files to the
/etc/node_exporter directory:
sudo cp node_exporter.* /etc/node_exporter
Copied!
Afterward, change the user and group ownership on the new directories and files
to nodeuser.
Then copy the node_exporter.crt file to the first server in any way you can.
For example, you can show the contents of a file and copy it to the clipboard:
Also, on both servers, you will need the htpasswd utility, which is part of
the apache2-utils package.
sudo apt install apache2-utils
Copied!
This concludes the preliminary preparation and we can proceed to installing
Prometheus on the first server in the next step.
Step 3 — Downloading and installing Prometheus
Prometheus cannot be installed from the default Ubuntu repositories at the time
of writing, so you need to get an archive of the current version. Go to the
official download page and copy
the link to the Linux package of the latest stable release.
Then log in to your first Linux server and download the archive with wget:
Once the archive is saved, we will verify the integrity of the download by
running the sha256sum program and compare its output with the corresponding
SHA256 Checksum value listed on the site:
Once you've confirmed that the value above matches the SHA256 Checksum on
the website, you can go ahead and unpack the archive using the command shown
below:
Prometheus is now installed on your server! Before starting it, let's take a
look at its configuration file in the next section.
Step 4 — Configuring Prometheus
In the previous step, we copied the sample config file to the /etc/prometheus
directory. Go ahead and open it in your text editor for examination.
sudo nano /etc/prometheus/prometheus.yml
Copied!
You should observe the following contents:
/etc/prometheus/prometheus.yml
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: "prometheus"
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ["localhost:9090"]
Copied!
This file uses the YAML format, which strictly forbids tabs and requires two
spaces for indentation so be careful when editing it. It contains some comments
to help you understand each setting. For example, under global settings, you
can change the default interval for scraping metrics from exporters.
/etc/prometheus/prometheus.yml
. . .
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
. . .
Copied!
The scrape_configs section contains all the exporters to collect metrics from.
By default, it is configured to collect data from the Prometheus itself:
/etc/prometheus/prometheus.yml
. . .
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: "prometheus"
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ["localhost:9090"]
Copied!
We won't be changing any settings for now until after setting up our Node
Exporter instance. In the next section, we will execute the prometheus CLI for
the first time.
Step 5 — Running the Prometheus CLI
First, we will start Prometheus manually, and then we will create a Systemd
service to manage it properly.
ts=2022-04-14T07:35:54.363Z caller=main.go:479 level=info msg="No time or size retention was set so using the default time retention" duration=15d
ts=2022-04-14T07:35:54.364Z caller=main.go:516 level=info msg="Starting Prometheus" version="(version=2.34.0, branch=HEAD, revision=881111fec4332c33094a6fb2680c71fffc427275)"
ts=2022-04-14T07:35:54.364Z caller=main.go:521 level=info build_context="(go=go1.17.8, [email protected], date=20220315-15:18:00)"
ts=2022-04-14T07:35:54.364Z caller=main.go:522 level=info host_details="(Linux 5.4.0-104-generic #118-Ubuntu SMP Wed Mar 2 19:02:41 UTC 2022 x86_64 ubuntu-2gb-nbg1-1 (none))"
ts=2022-04-14T07:35:54.365Z caller=main.go:523 level=info fd_limits="(soft=1024, hard=1048576)"
ts=2022-04-14T07:35:54.365Z caller=main.go:524 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2022-04-14T07:35:54.378Z caller=web.go:540 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
ts=2022-04-14T07:35:54.387Z caller=main.go:937 level=info msg="Starting TSDB ..."
ts=2022-04-14T07:35:54.394Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false
ts=2022-04-14T07:35:54.405Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
ts=2022-04-14T07:35:54.406Z caller=head.go:536 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=4.385µs
ts=2022-04-14T07:35:54.406Z caller=head.go:542 level=info component=tsdb msg="Replaying WAL, this may take a while"
ts=2022-04-14T07:35:54.407Z caller=head.go:613 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
ts=2022-04-14T07:35:54.407Z caller=head.go:619 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=49.735µs wal_replay_duration=1.364197ms total_replay_duration=1.547724ms
ts=2022-04-14T07:35:54.409Z caller=main.go:958 level=info fs_type=EXT4_SUPER_MAGIC
ts=2022-04-14T07:35:54.409Z caller=main.go:961 level=info msg="TSDB started"
ts=2022-04-14T07:35:54.409Z caller=main.go:1142 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
ts=2022-04-14T07:35:54.413Z caller=main.go:1179 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=3.377506ms db_storage=1.585µs remote_storage=2.659µs web_handler=718ns query_engine=8.708µs scrape=2.301903ms scrape_sd=31.007µs notify=386.697µs notify_sd=21.135µs rules=1.695µs tracing=11.188µs
ts=2022-04-14T07:35:54.413Z caller=main.go:910 level=info msg="Server is ready to receive web requests."
The last line above confirms that the server has started successfully, which
means our configuration is working correctly. However, starting the server
manually is not ideal so we will create a new systemd unit file so that it can
be managed through the Systemd daemon.
Kill the existing server with Ctrl-C, then create a prometheus.service file
in the /etc/systemd/system directory as shown below:
sudo nano /etc/systemd/system/prometheus.service
Copied!
Paste the following contents into the file and save it:
This service will perform a startup process similar to what we did manually
before. It will also automatically restart the service if any errors occur and
ensure the program is up and running after a system reboot.
Save and close the file, then reload the systemd configuration though the
command below:
sudo systemctl daemon-reload
Copied!
You can now start Prometheus by entering the following command:
Created symlink /etc/systemd/system/multi-user.target.wants/prometheus.service → /etc/systemd/system/prometheus.service.
At this point, you have Prometheus running on the first system. In the next
section, we will set up a Node Exporter instance on the second one.
Step 6 — Downloading and installing Node Exporter
Just like Prometheus, Node Exporter cannot be installed from the default Ubuntu
20.04 repositories, so you need to get an archive of the current version on the
Prometheus website. Copy the
link to the appropriate Linux package, then log in to your second Linux server
and download the archive:
You can also get the checksum of the downloaded file and compare it with the
SHA256 Checksum similar to the way we confirmed the Prometheus download in
step 3.
The Node Exporter CLI is now ready to go, but we need to secure it further so
that only authorized clients can connect to it.
Step 7 — Securing and running the Node Exporter instance
Before starting the Node Exporter, let's set up a secure connection. To do this,
we will utilize an SSL connections, as well as basic authorization. First,
generate a password hash through the htpasswd command:
htpasswd -nBC 10 "" | tr -d ':\n'
Copied!
This command will prompt you to enter and confirm the password. Enter something
you can remember, or use a password manager to generate one. For the purpose of
this tutorial, we used the password prom. As a result, you will get a string
like this:
We specified the path to the previously created certificate and the username and
the hashed password required to connect. For demonstration purposes, we also
used prom as the username.
Created symlink /etc/systemd/system/multi-user.target.wants/node_exporter.service → /etc/systemd/system/node_exporter.service.
Before adding the newly created exporter to our Prometheus config, let's check
that everything is working correctly. Return to the first server and use curl
to make a request to the second server:
where <your_second_server_ip> is the IP address of your second server.
You will observe some output that looks like like this:
Output
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 8
. . .
If you don't get a similar output, check that the firewall is not blocking
connections and also, check that the supplied username and password are correct.
Step 8 — Configuring Prometheus to Scrape Node Exporter
In this step, we will update the Prometheus settings to be able to collect
metrics from the Node Exporter. Staying on the first server, open the Prometheus
config file:
sudo nano /etc/prometheus/prometheus.yml
Copied!
Add the following lines to the end of the file under scrape_configs:
/etc/prometheus/prometheus.yml
. . .
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: "prometheus"
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ["localhost:9090"]
- job_name: 'node_exporter_client'
scrape_interval: 5s
scheme: https
tls_config:
ca_file: /etc/prometheus/node_exporter.crt
basic_auth:
username: prom
password: prom
static_configs:
- targets: ['<your_second_server_ip>:9100']
Copied!
The <your_second_server_ip> placeholder should be replaced with the IP address
of your second server. Save the file and restart the Prometheus server:
sudo systemctl restart prometheus
Copied!
Prometheus is now running and collecting information not only about itself but
also from the specified remote server.
Step 9 — Securing and working with the Prometheus web interface
Prometheus doesn't have any built-in authentication or authorization
capabilities, so to prevent unauthorized access to data in Prometheus, we will
use NGINX as a proxy. Ensure that it is installed
on your first server:
sudo apt install nginx
Copied!
If you are using a firewall you should also allow connections from NGINX:
sudo ufw allow 'Nginx HTTP'
Copied!
Now, using the htpasswd utility, create a password file. In this case, we will
use the username promadmin to access the web interface.
sudo htpasswd -c /etc/nginx/.htpasswd promadmin
Copied!
Next, you need to set up NGINX. For our tutorial, we will use the default
configuration file. Open it for editing:
sudo nano /etc/nginx/sites-enabled/default
Copied!
Find the location / block under the server block. It should look like this:
/etc/nginx/sites-enabled/default
. . .
location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ =404;
}
. . .
Save and close the file, then check your NGINX configuration:
sudo nginx -t
Copied!
The output should show that the syntax is ok and the test is successful:
Output
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
Afterward, reload the NGINX service:
sudo systemctl reload nginx
Copied!
Open your web browser and navigate to http://<your_first_server_ip>. In the
HTTP authentication dialogue box, enter the username and password you chose
previously in this step. After a successful login, you will see the Prometheus
UI:
You can verify the status of Node Explorer by clicking on the Status menu at
the top and then on the Targets menu option.
As you can see, all endpoints are active and there are no errors. If any error
occurs, its description will be visible in the Error column.
Conclusion
In this tutorial, you learned how to install and configure the Prometheus server
and Node Exporter. We also discussed creating secure connections between these
two components to ensure that no information is leaked to the outside world.