Loki Config With S3

Configuring Loki to use Amazon S3 as a storage backend involves updating the Loki configuration file to specify S3 as the object storage for your log data. Loki integrates with S3 to store logs efficiently, leveraging S3's scalable and durable storage.

Step-by-Step Guide to Configure Loki with S3

1. Prerequisites

• Loki Installation: Ensure Loki is installed and running. You can run Loki via Docker, Kubernetes, or as a standalone binary.
• Amazon S3 Bucket: Have an S3 bucket created and accessible with proper permissions.

2. Loki Configuration

You need to modify the Loki configuration file to use S3 for storage. Below is a sample configuration that sets up Loki to use S3.

Sample loki-config.yaml

auth_enabled: false

server:
  http_listen_port: 3100
  log_level: info

distributor:
  ring:
    kvstore:
      store: inmemory
    replication_factor: 1

ingester:
  chunk_idle_period: 5m
  chunk_target_size: 1048576
  max_chunk_age: 1h
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    heartbeat_period: 1m
  wal:
    enabled: true
    dir: /loki/wal

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/cache
    shared_store: s3
  s3:
    bucketnames: <your-bucket-name>
    endpoint: <your-s3-endpoint> # Use `s3.amazonaws.com` for AWS S3
    access_key_id: <your-access-key-id>
    secret_access_key: <your-secret-access-key>
    insecure: false # Set to true if not using SSL

compactor:
  working_directory: /loki/compactor
  shared_store: s3

query_range:
  split_queries_by_interval: 15m
  max_retries: 5
  parallelism: 32

frontend:
  max_outstanding_per_tenant: 1000
  max_outstanding_per_request: 500

3. Explanation of Configuration

• auth_enabled: Set to false if you are not using authentication; otherwise, configure authentication as needed.
• server: Configures the HTTP server that Loki will use to serve queries and receive logs.
• distributor: Manages log ingestion and distribution. Uses an in-memory key-value store for the ring.
• ingester: Handles log ingestion and chunk management. Stores logs in the specified directory.
• storage_config: Configures Loki to use S3 for log storage.
  • boltdb_shipper: Manages index files and cache.
  • s3: Specifies S3 settings including bucket name, endpoint, and credentials.
• compactor: Manages log compaction and storage in S3.
• query_range: Configuration related to querying logs.
• frontend: Configures frontend settings for querying.

4. Running Loki with S3

If using Docker, you can run Loki with the above configuration file mounted:

docker run -d --name=loki \\
  -v /path/to/loki-config.yaml:/etc/loki/loki-config.yaml \\
  grafana/loki:latest \\
  -config.file=/etc/loki/loki-config.yaml

If using Kubernetes, you can apply this configuration through a ConfigMap and refer to it in your Loki deployment.

5. IAM Permissions for S3

Ensure that the IAM role or user associated with the access_key_id and secret_access_key has appropriate permissions to access and manage the S3 bucket. The necessary permissions include:

• s3:PutObject
• s3:GetObject
• s3:ListBucket
• s3:DeleteObject

6. Testing and Verification

1. Verify Loki Logs: Check the Loki logs to ensure it starts correctly and connects to S3:

    

   Copied!

   docker logs loki
   

2. Check S3 Bucket: Verify that logs are being stored in your S3 bucket.

3. Query Logs: Use Grafana or another Loki client to query and view logs to confirm that Loki is correctly ingesting and retrieving logs from S3.

Additional Considerations

• Backup and Recovery: Regularly back up your S3 bucket and ensure you have a recovery plan.
• Monitoring: Monitor Loki and S3 for any issues related to performance or connectivity.

Got an article suggestion? Let us know

Explore more

Fluentd

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.