# Collect database and web server metrics

[Better Stack collector](https://betterstack.com/docs/logs/collector/) automatically recognizes databases and web servers running on your hosts and starts collecting their metrics: **PostgreSQL, MySQL, Redis, Memcached, MongoDB, Elasticsearch, Kafka, Apache, Nginx and Traefik**. Each recognized database comes with a pre-built dashboard covering database health, slow queries, connections, and queries per second.

Most services need no setup. MySQL and PostgreSQL need a monitoring user, and nginx, Apache and Traefik a status endpoint, before all their metrics arrive.

## Databases

### MySQL

Create a database user with the following permissions to collect MySQL database metrics:

```sql
[label Set up MySQL database user]
CREATE USER 'betterstack'@'%' IDENTIFIED BY '<PASSWORD>';
GRANT SELECT, PROCESS, REPLICATION CLIENT ON *.* TO 'betterstack'@'%';
```

### PostgreSQL

If the default `postgres` database does not exist, create it:

```sql
[label Create default Postgres database]
CREATE DATABASE postgres;
```

Create a database user with the `pg_monitor` role and enable the `pg_stat_statements` extension:

```sql
[label Set up Postgres database user]
CREATE ROLE betterstack WITH LOGIN PASSWORD '<PASSWORD>';
GRANT pg_monitor TO betterstack;
GRANT CONNECT ON DATABASE postgres TO betterstack;
GRANT USAGE ON SCHEMA public TO betterstack;
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
```

On hardened or RDS databases the default `public` schema grants are often revoked, so the `GRANT USAGE ON SCHEMA public` statement above is required.

Finally, load `pg_stat_statements` at startup and restart PostgreSQL. Amazon RDS already loads it, so you're done there.

[code-tabs]
```conf
[label postgresql.conf]
# Keep libraries already listed, e.g. 'pgaudit,pg_stat_statements'
shared_preload_libraries = 'pg_stat_statements'
```
```yaml
[label Docker Compose]
# Apply with docker compose up -d
services:
  postgres:
    command: postgres -c shared_preload_libraries=pg_stat_statements
```
```yaml
[label Bitnami Helm chart]
postgresqlSharedPreloadLibraries: "pgaudit,pg_stat_statements"
```
```yaml
[label CloudNativePG]
spec:
  postgresql:
    parameters:
      # Any pg_stat_statements.* parameter makes CloudNativePG load the library and restart
      pg_stat_statements.track: top
```
[/code-tabs]

![CleanShot 2026-02-07 at 7 .06.57.png](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/2470b6b4-846e-4ca1-bb26-fb86b4c41f00/lg2x =3244x2676)

## Web servers

Enable a status endpoint on nginx, Apache or Traefik so the collector can scrape its metrics.

### Nginx

The collector scrapes nginx metrics from the [stub_status](https://nginx.org/en/docs/http/ngx_http_stub_status_module.html) module at `/nginx_status`. Add the following to your nginx configuration:

```yaml
[label /etc/nginx/conf.d/status.conf]
server {
    listen 80;
    location /nginx_status {
        stub_status;
        allow 127.0.0.1;
        allow ::1;
        deny all;
    }
}
```

Reload nginx to apply the change:

```bash
[label Reload nginx]
nginx -s reload
```

[info]
#### Is the `stub_status` module available?
It is included in most nginx packages by default. Verify it with `nginx -V 2>&1 | grep -o with-http_stub_status_module`.
[/info]

### Apache

The collector scrapes Apache metrics from [mod_status](https://httpd.apache.org/docs/current/mod/mod_status.html) at `/server-status?auto`. Enable the module and add the following configuration:

```xml
[label /etc/apache2/mods-enabled/status.conf]
<Location "/server-status">
    SetHandler server-status
    Require local
</Location>
```

Enable the module and reload Apache:

[code-tabs]
```bash
[label Debian / Ubuntu]
a2enmod status
systemctl reload apache2
```
```bash
[label RHEL / CentOS / Amazon Linux]
# mod_status is usually enabled by default
# Verify with: httpd -M | grep status
systemctl reload httpd
```
[/code-tabs]

[info]
#### Running older Apache?
Set `ExtendedStatus On` to get the full set of metrics. On Apache 2.3.6 and up, this is already enabled by default.
[/info]

### Traefik

Enable the Prometheus exporter on Traefik:

[code-tabs]
```yaml
[label With yaml]
entryPoints:
  metrics:
    address: ":8082"

metrics:
  prometheus:
    entryPoint: metrics
    addEntryPointsLabels: true
    addRoutersLabels: true
    addServicesLabels: true
```
```bash
[label With CLI flags]
--entryPoints.metrics.address=:8082
--metrics.prometheus.entryPoint=metrics
```
[/code-tabs]

### Verify the web server configuration

After enabling the status endpoint, verify it's reachable from the host:

[code-tabs]
```bash
[label Nginx]
curl http://127.0.0.1/nginx_status
```
```bash
[label Apache]
curl http://127.0.0.1/server-status?auto
```
```bash
[label Traefik]
curl http://localhost:8082/metrics
```
[/code-tabs]

The process status in Better Stack will automatically change from **Configuration required** to **Active** within a few minutes.

## Kafka

The collector automatically discovers your Kafka brokers and collects cluster metadata with no Kafka configuration: broker count, partitions per topic, in-sync replicas (ISR), leader status and under-replication status per partition. These metrics power the **Overview** and **Partitions & replication** sections of the [Kafka dashboard](https://betterstack.com/dashboards/kafka ";_blank")

![Kafka dashboard](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/214d7120-547c-4eed-6ed4-f8bd82a4db00/lg2x =4000x2056)

### Basic Kafka metrics in Kubernetes

Brokers are discovered and reached automatically through the cluster network. Advertise each broker with a fully qualified domain name the cluster can resolve, such as the broker's Kubernetes Service DNS name:

```yaml
[label Advertise the broker FQDN]
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka.otel-demo.svc.cluster.local:9092
```

A bare hostname or `localhost` only resolves inside the broker's own pod, so the collector and other clients elsewhere in the cluster can't reach it.

### Basic Kafka metrics in Docker Compose

The collector connects to brokers from the host network. Publish the broker port and advertise a listener host clients can follow:

```yaml
[label docker-compose.yml]
services:
  kafka:
    image: apache/kafka:4.2.1
    ports:
      - "9092:9092"
    environment:
      KAFKA_LISTENERS: PLAINTEXT_HOST://0.0.0.0:9092,PLAINTEXT://0.0.0.0:19092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT_HOST://localhost:9092,PLAINTEXT://kafka:19092
```

Containers on the compose network keep connecting to `kafka:19092`. Then add the broker in Better Stack:

1. Go to [Sources](https://telemetry.betterstack.com/team/0/sources ";_blank") → your collector → **Configure** → **Collect metrics**.
2. Click **Collect metrics** and select **Kafka**.
3. Select the host running Kafka in **Collector host**.
4. Set **Service name** to match your Kafka service, like `kafka`.
5. Set **Host** to `localhost` and **Port** to `9092`. Use the Kafka protocol port, not the JMX exporter port.

## Get full Kafka metrics with JMX exporter

Kafka keeps broker-level performance metrics in JMX (Java Management Extensions) and doesn't expose them by default. Deploy the Prometheus JMX exporter as a Java agent on each broker to light up the rest of the [Kafka dashboard](https://betterstack.com/dashboards/kafka ";_blank"):

- Bytes and messages in/out, per topic
- Active controller, broker count, offline partitions
- Leader count, partition count, ISR changes
- Request rates per request type
- Log size per topic and partition

The setup is the same everywhere: put the agent JAR and a rules file on each broker, add the `-javaagent` flag to Kafka's JVM options, and let the collector scrape port `7071`.

### Create the rules file

Save the following configuration next to where the agent JAR will live:

```yaml
[label jmx-kafka-config.yaml]
lowercaseOutputName: true
lowercaseOutputLabelNames: true
rules:
# Per-topic broker throughput (BytesInPerSec, MessagesInPerSec, ...)
- pattern: kafka.server<type=(.+), name=(.+), topic=(.+)><>Count
  name: kafka_server_$1_$2_total
  type: COUNTER
  labels:
    topic: "$3"
# Per-client, per-partition gauges
- pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
  name: kafka_server_$1_$2
  type: GAUGE
  labels:
    clientId: "$3"
    topic: "$4"
    partition: "$5"
- pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
  name: kafka_server_$1_$2
  type: GAUGE
  labels:
    clientId: "$3"
    broker: "$4:$5"
# Log size per topic-partition
- pattern: kafka.log<type=(.+), name=(.+), topic=(.+), partition=(.*)><>Value
  name: kafka_log_$1_$2
  type: GAUGE
  labels:
    topic: "$3"
    partition: "$4"
# Network request counters
- pattern: kafka.network<type=(.+), name=(.+), request=(.+)><>Count
  name: kafka_network_$1_$2_total
  type: COUNTER
  labels:
    request: "$3"
# Catch-alls with a single extra mbean property, kept as a label
- pattern: kafka.server<type=(.+), name=(.+), (.+)=(.+)><>Value
  name: kafka_server_$1_$2
  type: GAUGE
  labels:
    $3: "$4"
- pattern: kafka.server<type=(.+), name=(.+), (.+)=(.+)><>Count
  name: kafka_server_$1_$2_total
  type: COUNTER
  labels:
    $3: "$4"
# Generic gauges and counters
- pattern: kafka.server<type=(.+), name=(.+)><>Value
  name: kafka_server_$1_$2
  type: GAUGE
- pattern: kafka.server<type=(.+), name=(.+)><>Count
  name: kafka_server_$1_$2_total
  type: COUNTER
- pattern: kafka.controller<type=(.+), name=(.+)><>Value
  name: kafka_controller_$1_$2
  type: GAUGE
- pattern: kafka.network<type=(.+), name=(.+)><>Value
  name: kafka_network_$1_$2
  type: GAUGE
- pattern: kafka.log<type=(.+), name=(.+)><>Value
  name: kafka_log_$1_$2
  type: GAUGE
```

Rules are applied first-match-wins. Keep the specific patterns above the generic catch-alls, otherwise per-topic metric names get mangled.

### JMX exporter in Kubernetes

Fetch the agent in an init container and attach it to the broker container. Mount the rules file to `/jmx-exporter/jmx-kafka-config.yaml`, for example from a ConfigMap:

```yaml
[label Kafka pod template]
initContainers:
  - name: jmx-exporter
    image: curlimages/curl:8.14.1
    command: ["curl", "-sSL", "-o", "/jmx-exporter/jmx_prometheus_javaagent.jar",
              "https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/1.0.1/jmx_prometheus_javaagent-1.0.1.jar"]
    volumeMounts:
      - name: jmx-exporter
        mountPath: /jmx-exporter
containers:
  - name: kafka
    env:
      - name: KAFKA_OPTS
        value: "-javaagent:/jmx-exporter/jmx_prometheus_javaagent.jar=7071:/jmx-exporter/jmx-kafka-config.yaml"
    ports:
      - containerPort: 7071
    volumeMounts:
      - name: jmx-exporter
        mountPath: /jmx-exporter
```

Annotate the Kafka pod template and the collector discovers and scrapes the endpoint automatically:

```yaml
[label Kafka pod template]
annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "7071"
  prometheus.io/path: "/metrics"
```

### JMX exporter in Docker Compose

Download the agent JAR next to the rules file:

```bash
[label Download JMX exporter agent]
curl -sSL -o ./jmx-exporter/jmx_prometheus_javaagent.jar \
  https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/1.0.1/jmx_prometheus_javaagent-1.0.1.jar
```

Mount the folder, attach the agent, and publish the metrics port:

```yaml
[label docker-compose.yml]
services:
  kafka:
    image: apache/kafka:4.2.1
    ports:
      - "7071:7071" # JMX exporter Prometheus endpoint
    environment:
      KAFKA_OPTS: >-
        -javaagent:/opt/jmx-exporter/jmx_prometheus_javaagent.jar=7071:/opt/jmx-exporter/jmx-kafka-config.yaml
    volumes:
      - ./jmx-exporter:/opt/jmx-exporter:ro
```

Restart Kafka and verify the endpoint:

```bash
[label Verify the metrics endpoint]
curl -s http://localhost:7071/metrics | grep kafka_server
```

Then add the endpoint as a scrape target in Better Stack:

1. Go to [Sources](https://telemetry.betterstack.com/team/0/sources ";_blank") → your collector → **Configure** → **Collect metrics**.
2. Click **Collect metrics** and select **Custom Prometheus exporter or service**.
3. Select the host running Kafka in **Collector host**.
4. Set **Service name** to match your Kafka service, like `kafka`. The shared name groups both metric sources under one service.
5. Set **Endpoint** to `http://localhost:7071/metrics`.

The collector scrapes the endpoint from the host network every 30 seconds. That's why port `7071` is published.

### Beware of Kafka CLI tools inheriting KAFKA_OPTS

Kafka CLI tools, including Docker healthchecks and Kubernetes probes calling scripts like `kafka-broker-api-versions.sh`, inherit `KAFKA_OPTS` and crash trying to attach a second agent to the busy port. Clear the variable for CLI invocations:

```bash
[label Healthcheck without the agent]
KAFKA_OPTS="" /opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092
```

### Verify the Kafka metrics

Within a few minutes, metrics appear in your collector source, and the scrape target shows as **Active** in your collector's **Configure** → **Collect metrics**. Check out the [Kafka dashboard](https://betterstack.com/dashboards/kafka ";_blank") to see your broker throughput, controller state, partition health, and storage in one place.

## Need help?

Please let us know at hello@betterstack.com.  
We're happy to help! 🙏
