How to Increase Prometheus Storage Retention

If you're using Prometheus for monitoring, you've probably encountered its storage limitations. By default, Prometheus only keeps your metrics for 15 days. This works fine when you're just starting out, but quickly becomes a problem when looking at long-term trends or meeting compliance requirements.

This guide will show you exactly how to increase your Prometheus retention period, so you can keep your valuable monitoring data for weeks, months, or even years.

Understanding Prometheus storage retention

Prometheus stores all your metrics in a time-series database on disk. The retention setting simply controls how long Prometheus keeps this data before automatically deleting it to free up space.

Prometheus controls data retention with two main settings:

--storage.tsdb.retention.time    # How long to keep your data
--storage.tsdb.retention.size    # Maximum size allowed for storage

The default is 15 days, regardless of size. When you hit either the time or size limit, Prometheus starts removing older data to make room for new metrics. This works for immediate needs but falls short when you need historical data.

Why you should increase Prometheus retention

Keeping your metrics longer gives you several significant benefits. You can spot gradual performance trends you'd miss with just two weeks of data. You'll make better infrastructure decisions based on actual historical usage.

You can find weekly, monthly, or seasonal patterns in your systems' behavior. If you have compliance requirements, longer retention helps you meet them. And when troubleshooting complex issues, having months of historical context enables you to find the root cause faster.

For example, a memory leak that increases usage by just 1% per day might seem harmless in a two-week window, but over 60 days, you'd see it's headed for a serious problem.

Checking your current retention settings

Before making changes, check your current setup. The fastest way is to query the Prometheus API:

curl -s http://localhost:9090/api/v1/status/runtimeinfo | grep retention

"storageRetention": "15d"

You can also check your Prometheus config file or command-line arguments:

ps aux | grep prometheus | grep retention

Note that these commands might return empty results if you haven't explicitly set retention options. This doesn't mean there's an error – it just indicates Prometheus is using its default 15-day retention period.

Or visit the Prometheus web interface and go to Status > Configuration to see all current settings:

Even in the web interface, you might not see storage settings listed if you haven't configured them explicitly. In this case, Prometheus is using the default values.

Configuring Prometheus for longer metric retention

Prometheus doesn’t provide a central configuration file for retention settings—instead, these flags need to be passed directly to the Prometheus process. How you do that depends on how Prometheus is deployed.

Below, we’ll walk through how to increase retention time in three common deployment environments: systemd-based setups, Docker Compose, and Kubernetes.

Each method involves passing the --storage.tsdb.retention.time flag to your Prometheus process with your desired retention value (e.g. 90d for 90 days).

Method 1: Modifying the Prometheus service file

If you installed Prometheus directly on a server and manage it with systemd, you’ll need to update the systemd service definition to include the new retention time.

Open the Prometheus service file:

sudo nano /etc/systemd/system/prometheus.service

Find the ExecStart line and add or update the retention flag:

[Service]
ExecStart=/usr/local/bin/prometheus \
    --config.file=/etc/prometheus/prometheus.yml \
    --storage.tsdb.path=/var/lib/prometheus \
    --storage.tsdb.retention.time=90d

Save the file and reload systemd:

sudo systemctl daemon-reload

Restart Prometheus:

sudo systemctl restart prometheus

Verify your changes took effect:

curl -s http://localhost:9090/api/v1/status/runtimeinfo | grep retention

"storageRetention": "90d"

Method 2: Updating Docker Compose configuration

If you’re running Prometheus using Docker Compose, you must update the command section in your docker-compose.yml file to pass the retention time flag.

Open your docker-compose.yml file:

nano docker-compose.yml

Update the Prometheus service section:

services:
  prometheus:
    image: prom/prometheus:v2.48.0
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=90d'
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    restart: unless-stopped

volumes:
  prometheus_data:

Apply the changes:

docker-compose down

docker-compose up -d

Check that it worked:

docker-compose exec prometheus prometheus --help | grep retention

Method 3: Kubernetes ConfigMap/deployment

If you're running Prometheus in Kubernetes, the retention time must be passed as an argument to the Prometheus container—usually within a StatefulSet or Deployment managed by your monitoring stack (like kube-prometheus-stack).

Edit your Prometheus StatefulSet:

kubectl edit statefulset prometheus -n monitoring

Add the retention flag to the container args:

spec:
  containers:
  - name: prometheus
    args:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=90d'

Save changes and Kubernetes will automatically update your pods.

Verify the new settings:

kubectl exec -it prometheus-0 -n monitoring -- \
  curl -s http://localhost:9090/api/v1/status/runtimeinfo | grep retention

Storage considerations for extended retention

When you increase retention, you need more disk space. Here's how to figure out how much you'll need.

Estimating storage requirements

You can estimate your storage needs with this simple formula:

Daily Storage = Ingestion Rate × 24 hours × Bytes per Sample

For example, if you collect 10,000 samples per second at around 2 bytes per sample (compressed):

10,000 samples/sec × 86,400 seconds × 2 bytes = ~1.7 GB per day

For 90 days of retention, you'd need approximately:

1.7 GB × 90 days = 153 GB of storage

To check your actual ingestion rate:

curl -s "http://localhost:9090/api/v1/query?query=rate(prometheus_tsdb_head_samples_appended_total[1h])" | jq

Optimizing storage efficiency

To maximize storage efficiency with longer retention:

Enable TSDB compression to reduce disk usage by 50-70%:

--storage.tsdb.wal-compression

Create recording rules to pre-aggregate metrics that you frequently query. This reduces the number of raw time series you need to store.

Consider increasing the scrape interval for metrics that don't need high-resolution sampling.

Set up an alert for when storage is running low:

- alert: PrometheusStorageFilling
  expr: (predict_linear(prometheus_tsdb_storage_blocks_bytes[1h], 24 * 3600) / prometheus_tsdb_storage_blocks_bytes_total) > 0.8
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "Prometheus storage is filling up"
    description: "Prometheus storage is predicted to fill within 24 hours."

Implementing remote storage for long-term retention

If you need to keep metrics for months or years, consider using remote storage options.

While solutions like Thanos, or Corter can handle years of data, Better Stack offers a more straightforward approach to remote storage with excellent performance.

Better Stack's Telemetry platform makes setting up remote storage for your Prometheus metrics remarkably easy. You can forward metrics from your existing Prometheus instance using Vector with Prometheus Remote-Write.

First, install Vector on your system:

bash -c "$(curl -L https://setup.vector.dev)"

apt-get install vector

Configure Vector to receive Prometheus metrics and forward them to Better Stack:

sources:
  better_stack_prometheus_source_$SOURCE_TOKEN:
    type: "prometheus_remote_write"
    address: "0.0.0.0:9090"

...
sinks:
  better_stack_http_sink_$SOURCE_TOKEN:
    type: "http"
    inputs:
      - "better_stack_prometheus_transform_$SOURCE_TOKEN"
    uri: "https://$INGESTING_HOST/metrics"
    method: "post"
    encoding:
      codec: "json"
    compression: "gzip"
    auth:
      strategy: "bearer"
      token: "$SOURCE_TOKEN"

Update your Prometheus configuration to write to Vector:

remote_write:
  - url: 'http://localhost:9090'

Restart Vector and reload Prometheus:

systemctl restart vector

ps -A | grep prometheus | tr -s ' ' | xargs | cut -f1 -d' ' | xargs kill -HUP

This approach lets you keep your recent data in Prometheus for fast local queries while sending everything to Better Stack for long-term storage and analysis. Better Stack handles all the complexities of storage optimization, downsampling, and query efficiency, so you don't have to manage a complex storage infrastructure.

Unlike traditional setups with Thanos that require configuring object storage like S3:

type: S3
config:
  bucket: "thanos"
  endpoint: "minio:9000"
  access_key: "access-key"
  secret_key: "secret-key"
  insecure: true

Better Stack eliminates this complexity with a fully managed solution that scales with your needs.

Troubleshooting common retention issues

Here are solutions to common problems you might face when increasing retention.

Disk usage growing too fast

If your disk usage shoots up after increasing retention:

df -h /var/lib/prometheus

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       100G   85G   15G  85% /

Solution: Set up monitoring to predict storage growth and automatically expand volumes before they fill up.

Slow query performance

If queries covering months of data become painfully slow:

time curl -s "http://localhost:9090/api/v1/query?query=rate(node_cpu_seconds_total[30d])"

real    0m3.452s
user    0m0.004s
sys     0m0.004s

Solution: Create recording rules for common queries and consider using remote storage with downsampling for historical data.

Failed compaction

If you see storage compaction errors in logs:

level=error ts=2023-06-15T14:22:00Z msg="Compaction failed" err="write: no space left on device"

Solution: Ensure you have at least 30% free space on your storage volume for TSDB compaction to work properly.

Best practices for Prometheus retention management

Managing retention isn't just about setting a number of days—it's about balancing performance, cost, and usability. These proven practices help keep your Prometheus setup efficient, reliable, and ready for long-term insights:

• Use a tiered retention strategy: short-term in local Prometheus, medium-term in remote storage, long-term with downsampling
  
• Monitor disk usage growth and set alerts for low storage
  
• Check query performance across different time ranges
  
• Identify and reduce high-cardinality metrics
  
• Remove metrics that are no longer needed
  
• Adjust retention periods based on real usage patterns
  
• Document your chosen retention periods and the reasons behind them
  
• Keep a changelog of retention and storage configuration updates
  
• Write clear instructions for expanding storage or troubleshooting retention issues
  

Final thoughts

Increasing Prometheus retention gives you powerful insights from longer-term data analysis. It helps you spot trends, plan capacity more effectively, and troubleshoot complex issues with better context.

Your retention strategy should match your specific needs, compliance requirements, and available resources. Whether you choose to extend local storage or implement remote storage, the key is finding the right balance between retention length, system performance, and cost.

For very large deployments or extremely long retention requirements, consider specialized solutions like Better Stack that build on Prometheus while addressing the challenges of long-term storage and querying massive time-series datasets.

Got an article suggestion? Let us know

Next article

Deploying Prometheus on Kubernetes using Helm charts

Learn how to deploy Prometheus on Kubernetes using Helm charts, configure custom targets, and set up advanced monitoring capabilities for containerized applications

→

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.