Collect database and web server metrics

Better Stack collector automatically recognizes databases and web servers running on your hosts and starts collecting their metrics: PostgreSQL, MySQL, Redis, Memcached, MongoDB, Elasticsearch, Kafka, Apache, Nginx and Traefik. Each recognized database comes with a pre-built dashboard covering database health, slow queries, connections, and queries per second.

Most services need no setup. MySQL and PostgreSQL need a monitoring user, and nginx, Apache and Traefik a status endpoint, before all their metrics arrive.

Databases

MySQL

Create a database user with the following permissions to collect MySQL database metrics:

Set up MySQL database user
CREATE USER 'betterstack'@'%' IDENTIFIED BY '<PASSWORD>';
GRANT SELECT, PROCESS, REPLICATION CLIENT ON *.* TO 'betterstack'@'%';

PostgreSQL

If the default postgres database does not exist, create it:

Create default Postgres database
CREATE DATABASE postgres;

Create a database user with the pg_monitor role and enable the pg_stat_statements extension:

Set up Postgres database user
CREATE ROLE betterstack WITH LOGIN PASSWORD '<PASSWORD>';
GRANT pg_monitor TO betterstack;
GRANT CONNECT ON DATABASE postgres TO betterstack;
GRANT USAGE ON SCHEMA public TO betterstack;
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

On hardened or RDS databases the default public schema grants are often revoked, so the GRANT USAGE ON SCHEMA public statement above is required.

Finally, load pg_stat_statements at startup and restart PostgreSQL. Amazon RDS already loads it, so you're done there.

postgresql.conf Docker Compose Bitnami Helm chart CloudNativePG
# Keep libraries already listed, e.g. 'pgaudit,pg_stat_statements'
shared_preload_libraries = 'pg_stat_statements'
# Apply with docker compose up -d
services:
  postgres:
    command: postgres -c shared_preload_libraries=pg_stat_statements
postgresqlSharedPreloadLibraries: "pgaudit,pg_stat_statements"
spec:
  postgresql:
    parameters:
      # Any pg_stat_statements.* parameter makes CloudNativePG load the library and restart
      pg_stat_statements.track: top

CleanShot 2026-02-07 at 7 .06.57.png

Web servers

Enable a status endpoint on nginx, Apache or Traefik so the collector can scrape its metrics.

Nginx

The collector scrapes nginx metrics from the stub_status module at /nginx_status. Add the following to your nginx configuration:

/etc/nginx/conf.d/status.conf
server {
    listen 80;
    location /nginx_status {
        stub_status;
        allow 127.0.0.1;
        allow ::1;
        deny all;
    }
}

Reload nginx to apply the change:

Reload nginx
nginx -s reload

Is the stub_status module available?

It is included in most nginx packages by default. Verify it with nginx -V 2>&1 | grep -o with-http_stub_status_module.

Apache

The collector scrapes Apache metrics from mod_status at /server-status?auto. Enable the module and add the following configuration:

/etc/apache2/mods-enabled/status.conf
<Location "/server-status">
    SetHandler server-status
    Require local
</Location>

Enable the module and reload Apache:

Debian / Ubuntu RHEL / CentOS / Amazon Linux
a2enmod status
systemctl reload apache2
# mod_status is usually enabled by default
# Verify with: httpd -M | grep status
systemctl reload httpd

Running older Apache?

Set ExtendedStatus On to get the full set of metrics. On Apache 2.3.6 and up, this is already enabled by default.

Traefik

Enable the Prometheus exporter on Traefik:

With yaml With CLI flags
entryPoints:
  metrics:
    address: ":8082"

metrics:
  prometheus:
    entryPoint: metrics
    addEntryPointsLabels: true
    addRoutersLabels: true
    addServicesLabels: true
--entryPoints.metrics.address=:8082
--metrics.prometheus.entryPoint=metrics

Verify the web server configuration

After enabling the status endpoint, verify it's reachable from the host:

Nginx Apache Traefik
curl http://127.0.0.1/nginx_status
curl http://127.0.0.1/server-status?auto
curl http://localhost:8082/metrics

The process status in Better Stack will automatically change from Configuration required to Active within a few minutes.

Kafka

The collector automatically discovers your Kafka brokers and collects cluster metadata with no Kafka configuration: broker count, partitions per topic, in-sync replicas (ISR), leader status and under-replication status per partition. These metrics power the Overview and Partitions & replication sections of the Kafka dashboard

Kafka dashboard

Basic Kafka metrics in Kubernetes

Brokers are discovered and reached automatically through the cluster network. Advertise each broker with a fully qualified domain name the cluster can resolve, such as the broker's Kubernetes Service DNS name:

Advertise the broker FQDN
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka.otel-demo.svc.cluster.local:9092

A bare hostname or localhost only resolves inside the broker's own pod, so the collector and other clients elsewhere in the cluster can't reach it.

Basic Kafka metrics in Docker Compose

The collector connects to brokers from the host network. Publish the broker port and advertise a listener host clients can follow:

docker-compose.yml
services:
  kafka:
    image: apache/kafka:4.2.1
    ports:
      - "9092:9092"
    environment:
      KAFKA_LISTENERS: PLAINTEXT_HOST://0.0.0.0:9092,PLAINTEXT://0.0.0.0:19092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT_HOST://localhost:9092,PLAINTEXT://kafka:19092

Containers on the compose network keep connecting to kafka:19092. Then add the broker in Better Stack:

  1. Go to Sources → your collector → Configure → Collect metrics.
  2. Click Collect metrics and select Kafka.
  3. Select the host running Kafka in Collector host.
  4. Set Service name to match your Kafka service, like kafka.
  5. Set Host to localhost and Port to 9092. Use the Kafka protocol port, not the JMX exporter port.

Get full Kafka metrics with JMX exporter

Kafka keeps broker-level performance metrics in JMX (Java Management Extensions) and doesn't expose them by default. Deploy the Prometheus JMX exporter as a Java agent on each broker to light up the rest of the Kafka dashboard:

  • Bytes and messages in/out, per topic
  • Active controller, broker count, offline partitions
  • Leader count, partition count, ISR changes
  • Request rates per request type
  • Log size per topic and partition

The setup is the same everywhere: put the agent JAR and a rules file on each broker, add the -javaagent flag to Kafka's JVM options, and let the collector scrape port 7071.

Create the rules file

Save the following configuration next to where the agent JAR will live:

jmx-kafka-config.yaml
lowercaseOutputName: true
lowercaseOutputLabelNames: true
rules:
# Per-topic broker throughput (BytesInPerSec, MessagesInPerSec, ...)
- pattern: kafka.server<type=(.+), name=(.+), topic=(.+)><>Count
  name: kafka_server_$1_$2_total
  type: COUNTER
  labels:
    topic: "$3"
# Per-client, per-partition gauges
- pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
  name: kafka_server_$1_$2
  type: GAUGE
  labels:
    clientId: "$3"
    topic: "$4"
    partition: "$5"
- pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
  name: kafka_server_$1_$2
  type: GAUGE
  labels:
    clientId: "$3"
    broker: "$4:$5"
# Log size per topic-partition
- pattern: kafka.log<type=(.+), name=(.+), topic=(.+), partition=(.*)><>Value
  name: kafka_log_$1_$2
  type: GAUGE
  labels:
    topic: "$3"
    partition: "$4"
# Network request counters
- pattern: kafka.network<type=(.+), name=(.+), request=(.+)><>Count
  name: kafka_network_$1_$2_total
  type: COUNTER
  labels:
    request: "$3"
# Catch-alls with a single extra mbean property, kept as a label
- pattern: kafka.server<type=(.+), name=(.+), (.+)=(.+)><>Value
  name: kafka_server_$1_$2
  type: GAUGE
  labels:
    $3: "$4"
- pattern: kafka.server<type=(.+), name=(.+), (.+)=(.+)><>Count
  name: kafka_server_$1_$2_total
  type: COUNTER
  labels:
    $3: "$4"
# Generic gauges and counters
- pattern: kafka.server<type=(.+), name=(.+)><>Value
  name: kafka_server_$1_$2
  type: GAUGE
- pattern: kafka.server<type=(.+), name=(.+)><>Count
  name: kafka_server_$1_$2_total
  type: COUNTER
- pattern: kafka.controller<type=(.+), name=(.+)><>Value
  name: kafka_controller_$1_$2
  type: GAUGE
- pattern: kafka.network<type=(.+), name=(.+)><>Value
  name: kafka_network_$1_$2
  type: GAUGE
- pattern: kafka.log<type=(.+), name=(.+)><>Value
  name: kafka_log_$1_$2
  type: GAUGE

Rules are applied first-match-wins. Keep the specific patterns above the generic catch-alls, otherwise per-topic metric names get mangled.

JMX exporter in Kubernetes

Fetch the agent in an init container and attach it to the broker container. Mount the rules file to /jmx-exporter/jmx-kafka-config.yaml, for example from a ConfigMap:

Kafka pod template
initContainers:
  - name: jmx-exporter
    image: curlimages/curl:8.14.1
    command: ["curl", "-sSL", "-o", "/jmx-exporter/jmx_prometheus_javaagent.jar",
              "https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/1.0.1/jmx_prometheus_javaagent-1.0.1.jar"]
    volumeMounts:
      - name: jmx-exporter
        mountPath: /jmx-exporter
containers:
  - name: kafka
    env:
      - name: KAFKA_OPTS
        value: "-javaagent:/jmx-exporter/jmx_prometheus_javaagent.jar=7071:/jmx-exporter/jmx-kafka-config.yaml"
    ports:
      - containerPort: 7071
    volumeMounts:
      - name: jmx-exporter
        mountPath: /jmx-exporter

Annotate the Kafka pod template and the collector discovers and scrapes the endpoint automatically:

Kafka pod template
annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "7071"
  prometheus.io/path: "/metrics"

JMX exporter in Docker Compose

Download the agent JAR next to the rules file:

Download JMX exporter agent
curl -sSL -o ./jmx-exporter/jmx_prometheus_javaagent.jar \
  https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/1.0.1/jmx_prometheus_javaagent-1.0.1.jar

Mount the folder, attach the agent, and publish the metrics port:

docker-compose.yml
services:
  kafka:
    image: apache/kafka:4.2.1
    ports:
      - "7071:7071" # JMX exporter Prometheus endpoint
    environment:
      KAFKA_OPTS: >-
        -javaagent:/opt/jmx-exporter/jmx_prometheus_javaagent.jar=7071:/opt/jmx-exporter/jmx-kafka-config.yaml
    volumes:
      - ./jmx-exporter:/opt/jmx-exporter:ro

Restart Kafka and verify the endpoint:

Verify the metrics endpoint
curl -s http://localhost:7071/metrics | grep kafka_server

Then add the endpoint as a scrape target in Better Stack:

  1. Go to Sources → your collector → Configure → Collect metrics.
  2. Click Collect metrics and select Custom Prometheus exporter or service.
  3. Select the host running Kafka in Collector host.
  4. Set Service name to match your Kafka service, like kafka. The shared name groups both metric sources under one service.
  5. Set Endpoint to http://localhost:7071/metrics.

The collector scrapes the endpoint from the host network every 30 seconds. That's why port 7071 is published.

Beware of Kafka CLI tools inheriting KAFKA_OPTS

Kafka CLI tools, including Docker healthchecks and Kubernetes probes calling scripts like kafka-broker-api-versions.sh, inherit KAFKA_OPTS and crash trying to attach a second agent to the busy port. Clear the variable for CLI invocations:

Healthcheck without the agent
KAFKA_OPTS="" /opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092

Verify the Kafka metrics

Within a few minutes, metrics appear in your collector source, and the scrape target shows as Active in your collector's Configure → Collect metrics. Check out the Kafka dashboard to see your broker throughput, controller state, partition health, and storage in one place.

Need help?

Please let us know at hello@betterstack.com.
We're happy to help! 🙏