Host events in your own bucket

You can store your warehouse events in a S3-compatible bucket you control. This is useful for:

• Having tight control over storing sensitive data.
• Retaining data beyond the standard retention period.
• Keeping a long-term archive of your data.

This feature is available for any S3-compatible service, such as AWS S3, CloudFlare R2, DigitalOcean Spaces, etc. We recommend using Cloudflare R2 since it doesn't charge for egress fees.

Set up your source

1. Go to Warehouse -> Sources -> Connect source.
2. After configuring basic information, scroll down to Advanced settings.
3. Choose your provider and fill in the bucket access details.
4. Create your source, and start sending your data 🚀

Data retention

By default, the data will be automatically removed from your bucket after the data retention.

You can choose to keep the data in the bucket after your retention period ends. This ensures you keep a permanent copy, even if events are no longer accessible in Better Stack.

While creating your source, enable the Keep data after retention option.

File format

Events are stored in your bucket are in the ClickHouse Native format. This enables highly efficient storage and querying.

You can inspect and query the files using the clickhouse-local tool.

Querying events with ClickHouse

To query your files directly from S3:

1. Install clickhouse-local for your OS.
2. Use the ClickHouse s3 table function to query your files.

./clickhouse local \
-q "SELECT dt, raw
    FROM s3(
      'https://<region>.s3-compatible-service.com/<bucket>/<path>/<file>',
      '<access-key-id>',
      '<access-key-secret>'
    )
    ORDER BY dt DESC
    LIMIT 500"

To find out the structure of the data in any file:

./clickhouse local \
-q "DESCRIBE s3(
      'https://<region>.s3-compatible-service.com/<bucket>/<path>/<file>',
      '<access-key-id>',
      '<access-key-secret>'
    )"

Downloading events for local analysis (Optional)

If you prefer to download and analyze your events locally:

1. Download any file from your bucket. No need to decompress the file - the native format is handled directly.
2. Run ./clickhouse local with your query, replacing FROM s3(...) with FROM file(...):

./clickhouse local \
-q "SELECT dt, raw
    FROM file('<downloaded-file>')
    ORDER BY dt DESC
    LIMIT 500"

Search your events

You can run any ClickHouse SQL expressions such as:

./clickhouse local \
-q "SELECT dt, raw
    FROM s3(
      'https://<region>.s3-compatible-service.com/<bucket>/<path>/<file>',
      '<access-key-id>',
      '<access-key-secret>'
    )
    WHERE raw ILIKE '%search term%'
    ORDER BY dt DESC
    LIMIT 500"

Export to CSV, JSON, etc

You can use any ClickHouse-supported format with --output-format.

./clickhouse local \
--output-format=JSONEachRow \
-q "SELECT dt, raw
    FROM s3(
      'https://<region>.s3-compatible-service.com/<bucket>/<path>/<file>',
      '<access-key-id>',
      '<access-key-secret>'
    )"

Read more about output formats in our Ad-hoc SQL API docs.