# Custom log group routing

Route specific AWS CloudWatch log groups **to separate Better Stack sources** with different retention periods.

By default, the [AWS CloudFormation integration](https://betterstack.com/docs/logs/aws-cloudformation/) forwards all CloudWatch log groups to a single Better Stack source. However, you may need different retention periods for different types of logs. For example:

- **Security and audit logs** (e.g. CloudTrail) with a longer retention period.
- **Application logs** with a shorter retention period.
- **Error logs** with a medium retention period.

You can achieve this by creating additional Better Stack sources with different retention settings and routing specific log groups to them using custom Amazon Data Firehose streams.

## Prerequisites

An existing [AWS CloudFormation integration](https://betterstack.com/docs/logs/aws-cloudformation/) deployed in your AWS account.

Access to the [AWS Console](https://console.aws.amazon.com/) with permissions to manage Firehose streams, IAM roles, and CloudWatch log subscriptions.

## 1. Disable the log group in your primary source

In your existing AWS source in Better Stack, disable log ingestion for the log group you want to route to a different source.

Go to **Telemetry** -> [Sources](https://telemetry.betterstack.com/team/0/sources ";_blank") -> your AWS source -> **Configure**.

Find the log group in the **AWS log groups** section, untick it, and press **Save**.

![Disable the log group in your primary source](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/4b972eca-30d1-48e3-8154-6b0e54454200/lg2x =2614x2134)

## 2. Create a new source

[Create a new HTTP source](https://telemetry.betterstack.com/team/0/sources/new?platform=http ";_blank") in Better Stack to hold the re-routed logs. This source can have a different retention period than your primary AWS source. Give it a descriptive name, e.g. `AWS CloudTrail logs`.

Take note of the **cluster name** and the **source token** — you'll need it in the next step.

![Create a new source](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/be54cd7b-442a-41f8-0654-e7a0644b0a00/orig =2614x1040)

## 3. Create a new Firehose stream

In the AWS Console, create a new Amazon Data Firehose stream that will deliver logs to your new Better Stack source.

Go to **Amazon Data Firehose** -> [Firehose streams](https://console.aws.amazon.com/firehose/home#/streams) -> **Create Firehose stream** and configure the following:

### Source and destination

- Source: **Direct PUT**
- Destination: **HTTP endpoint**
- Firehose stream name: use a name beginning with `better-stack-` (e.g. `better-stack-cloudtrail-logs`)

![Firehose stream creation](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/00657cb6-b2f7-45cf-1975-7468456b2800/md2x =2614x1394)

[info]
The `better-stack-` prefix is required for the stream name.
[/info]

### Transform source records

Enable **Transform source records with AWS Lambda**. Choose the existing Lambda function that was created by the Better Stack CloudFormation stack.

![Lambda transformation settings](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/baf69803-d6ee-48e6-bc14-b6a38efb6f00/md2x =2592x1288)

### Destination settings:

Set the **HTTP endpoint URL** depending on your new source and your AWS region.

[code-tabs]
```
[label Format of the HTTP endpoint URL]
https://<cluster>-aws-<aws-region>.betterstackdata.com/aws-firehose
```
```
[label Example]
https://us-east-9-aws-eu-central-1.betterstackdata.com/aws-firehose
```
[/code-tabs]

You can find your cluster name on the configuration page of your new Better Stack source.

As your **Access key**, paste the **source token** from your new Better Stack source.

Set **Content encoding** to **GZIP** to compress your data in transit.

![Firehose destination settings](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/d80c62f2-038f-48d8-0f16-8d8f7a6f1000/lg2x =2614x1534)

### Backup settings

Configure the backup S3 bucket to be the same as your existing Better Stack Firehose stream.

### Permissions

Choose the existing `better-stack-firehose` IAM role.

![Firehose permissions settings](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/78b52d8f-3e91-436b-84df-417fa8ea9c00/lg2x =2614x1672)

3. Press **Create Firehose stream**.

## 4. Update the IAM subscription role

The `better-stack-logs-subscription-role` IAM role **needs permission to write** to your new Firehose stream.

Go to [IAM -> Roles](https://console.aws.amazon.com/iam/home#/roles) and find the `better-stack-logs-subscription-role` role.

Edit the attached policy to **allow writing to Firehose streams** matching the `better-stack-` prefix, rather than just the default streams.

## 5. Add a CloudWatch subscription filter

Route your chosen log group to the new Firehose stream using a CloudWatch subscription filter.

1. Go to [CloudWatch -> Log groups](https://console.aws.amazon.com/cloudwatch/home#logsV2:log-groups).
2. Open the log group you want to route, e.g. your CloudTrail log group.
3. Switch to the **Subscription filters** tab.
4. Choose **Create** -> **Create Amazon Data Firehose subscription filter**.
5. Select your new Firehose stream as the destination, e.g. `better-stack-cloudtrail-logs`.
6. Optionally, specify a filter pattern to forward only a subset of logs from this log group.
7. Press **Start streaming**.

Your re-routed logs should now appear in your new Better Stack source in [Live tail](https://telemetry.betterstack.com/team/0/tail ";_blank").

## Routing multiple log groups

Repeat this process for as many log groups as you need. Each log group or set of log groups can be routed to a different Better Stack source with its own retention period.

You can reuse a single Firehose stream for multiple log groups that should share the same Better Stack source — simply add a subscription filter to each log group pointing to the same stream.

## Need help?

Please let us know at hello@betterstack.com.  
We're happy to help! 🙏