# Hosting data in your own bucket

You can store your telemetry data in a bucket you control. This is useful for:

- Keeping a long-term archive of your data.
- Having tight control over storing sensitive data.
- Retaining data beyond the standard retention period.

This feature is available for any **S3-compatible service**, such as AWS S3, CloudFlare R2, DigitalOcean Spaces, etc. We recommend using [Cloudflare R2](https://www.cloudflare.com/products/r2/) since it doesn't charge for egress fees.

[note]
This is a [billable feature](https://betterstack.com/pricing#logs ";_blank") that **cannot be changed** after a source is created.
[/note]

## Set up your source

1. Go to Telemetry -> Sources -> 
[Connect source](https://telemetry.betterstack.com/team/0/sources/new ";_blank").
2. After configuring basic information, **scroll down to Advanced settings**.
3. Choose your provider and **fill in the bucket access details**.
4. Create your source, and start sending your data 🚀

![Advanced settings in source creation](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/6a932883-20f5-44e3-4ebf-8623b365cd00/orig =3974x4658)

[note]
#### Unsure about how to configure your storage or need a custom setup?

Please let us know at hello@betterstack.com. We're happy to help! 🙏
[/note]

## Data retention

By default, the data will be automatically removed from your bucket after the logs data retention.

You can choose to keep the data in the bucket after your retention period ends. This ensures **you keep a permanent copy**, even if logs are no longer accessible in Better Stack.

While creating your source, enable the **Keep data after retention** option.

![Keep data after retention](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/f72a53d9-e209-485e-4f4b-a7de3ce85600/md1x =3736x1778)

## File format

The log files stored in your bucket are in the [ClickHouse Native format](https://clickhouse.com/docs/interfaces/formats/Native). This enables highly efficient storage and querying.

You can inspect and query the files using the [clickhouse-local](https://clickhouse.com/docs/en/operations/utilities/clickhouse-local) tool.

### Querying log files with ClickHouse

To query your log files directly from S3:

1. [Install clickhouse-local](https://clickhouse.com/docs/operations/utilities/clickhouse-local#download-clickhouse-local) for your OS.
2. Use the [ClickHouse s3 table function](https://clickhouse.com/docs/sql-reference/table-functions/s3) to query your files.

```bash
[label Query logs from S3 directly]
./clickhouse local \
-q "SELECT dt, raw
    FROM s3(
      'https://<region>.s3-compatible-service.com/<bucket>/<path>/<file>',
      '<access-key-id>',
      '<access-key-secret>'
    )
    ORDER BY dt DESC
    LIMIT 500"
```

To find out the structure of the data in any file:

```bash
[label Find out the structure of data in any file]
./clickhouse local \
-q "DESCRIBE s3(
      'https://<region>.s3-compatible-service.com/<bucket>/<path>/<file>',
      '<access-key-id>',
      '<access-key-secret>'
    )"
```

### Downloading log files for local analysis (Optional)

If you prefer to download and analyze your logs locally:

1. Download any file from your bucket. No need to decompress the file - the native format is handled directly.
2. Run `./clickhouse local` with your query, replacing `FROM s3(...)` with `FROM file(...)`:

```bash
[label Query downloaded logs using SQL]
./clickhouse local \
-q "SELECT dt, raw
    FROM file('<downloaded-file>')
    ORDER BY dt DESC
    LIMIT 500"
```

### Search your logs

You can use ClickHouse SQL as you would in [Explore](https://betterstack.com/docs/logs/using-logtail/explore-logs/).

```bash
[label Search logs containing "search term"]
./clickhouse local \
-q "SELECT dt, raw
    FROM s3(
      'https://<region>.s3-compatible-service.com/<bucket>/<path>/<file>',
      '<access-key-id>',
      '<access-key-secret>'
    )
    WHERE raw ILIKE '%search term%'
    ORDER BY dt DESC
    LIMIT 500"
```

#### Export to CSV, JSON, etc

You can use any ClickHouse-supported format with `--output-format`.

```bash
[label Export to ND-JSON]
./clickhouse local \
--output-format=JSONEachRow \
-q "SELECT dt, raw
    FROM s3(
      'https://<region>.s3-compatible-service.com/<bucket>/<path>/<file>',
      '<access-key-id>',
      '<access-key-secret>'
    )"
```

Read more about [output formats in our Connection API](https://betterstack.com/docs/logs/query-api/connect-remotely/#output-formats) docs.