Explore documentation
  1. Welcome to Telemetry 👋
  2. Quick start guide
  3. Wide events vs. metrics
  4. Ingesting data
    1. Better Stack collector
    2. Vector
    3. OpenTelemetry
    4. Ingesting via HTTP API
    5. Ingesting metrics
    6. Ingesting logs
      1. Docker & Kubernetes
      2. Operating systems
      3. Log forwarders
      4. Programming languages
      5. Cloud platforms
        1. AWS
          1. Getting started
          2. Custom log group routing
          3. Amazon CloudWatch
          4. Amazon ECS
          5. Amazon EKS
          6. AWS Elastic Load Balancing
          7. AWS Fargate
          8. AWS Lambda
          9. AWS Direct Connect
        2. Azure
        3. Cloudflare
        4. Google
        5. Digital Ocean
        6. Heroku
        7. Vercel
        8. Render
        9. Fly.io
        10. Supabase
      6. Web servers & Load balancers
      7. Databases & queues
      8. Storage
    7. Migrating to Better Stack
  5. Using the product
    1. AI SRE ↗
    2. MCP server ↗
    3. Transforming ingested data
    4. Querying data
    5. Tracing
    6. Alerts
  6. Telemetry API
    1. Getting an API token
    2. Sources & collectors
    3. Dashboards
    4. Explorations
    5. Alerts API
    6. Connections API
    7. Query API ↗
  7. Alternative to
    1. Lightstep

Custom log group routing

Route specific AWS CloudWatch log groups to separate Better Stack sources with different retention periods.

By default, the AWS CloudFormation integration forwards all CloudWatch log groups to a single Better Stack source. However, you may need different retention periods for different types of logs. For example:

  • Security and audit logs (e.g. CloudTrail) with a longer retention period.
  • Application logs with a shorter retention period.
  • Error logs with a medium retention period.

You can achieve this by creating additional Better Stack sources with different retention settings and routing specific log groups to them using custom Amazon Data Firehose streams.

Prerequisites

An existing AWS CloudFormation integration deployed in your AWS account.

Access to the AWS Console with permissions to manage Firehose streams, IAM roles, and CloudWatch log subscriptions.

1. Disable the log group in your primary source

In your existing AWS source in Better Stack, disable log ingestion for the log group you want to route to a different source.

Go to Telemetry -> Sources -> your AWS source -> Configure.

Find the log group in the AWS log groups section, untick it, and press Save.

Disable the log group in your primary source

2. Create a new source

Create a new HTTP source in Better Stack to hold the re-routed logs. This source can have a different retention period than your primary AWS source. Give it a descriptive name, e.g. AWS CloudTrail logs.

Take note of the cluster name and the source token — you'll need it in the next step.

Create a new source

3. Create a new Firehose stream

In the AWS Console, create a new Amazon Data Firehose stream that will deliver logs to your new Better Stack source.

Go to Amazon Data Firehose -> Firehose streams -> Create Firehose stream and configure the following:

Source and destination

  • Source: Direct PUT
  • Destination: HTTP endpoint
  • Firehose stream name: use a name beginning with better-stack- (e.g. better-stack-cloudtrail-logs)

Firehose stream creation

The better-stack- prefix is required for the stream name.

Transform source records

Enable Transform source records with AWS Lambda. Choose the existing Lambda function that was created by the Better Stack CloudFormation stack.

Lambda transformation settings

Destination settings:

Set the HTTP endpoint URL depending on your new source and your AWS region.

Format of the HTTP endpoint URL Example 
https://<cluster>-aws-<aws-region>.betterstackdata.com/aws-firehose
https://us-east-9-aws-eu-central-1.betterstackdata.com/aws-firehose

You can find your cluster name on the configuration page of your new Better Stack source.

As your Access key, paste the source token from your new Better Stack source.

Set Content encoding to GZIP to compress your data in transit.

Firehose destination settings

Backup settings

Configure the backup S3 bucket to be the same as your existing Better Stack Firehose stream.

Permissions

Choose the existing better-stack-firehose IAM role.

Firehose permissions settings

  1. Press Create Firehose stream.

4. Update the IAM subscription role

The better-stack-logs-subscription-role IAM role needs permission to write to your new Firehose stream.

Go to IAM -> Roles and find the better-stack-logs-subscription-role role.

Edit the attached policy to allow writing to Firehose streams matching the better-stack- prefix, rather than just the default streams.

5. Add a CloudWatch subscription filter

Route your chosen log group to the new Firehose stream using a CloudWatch subscription filter.

  1. Go to CloudWatch -> Log groups.
  2. Open the log group you want to route, e.g. your CloudTrail log group.
  3. Switch to the Subscription filters tab.
  4. Choose Create -> Create Amazon Data Firehose subscription filter.
  5. Select your new Firehose stream as the destination, e.g. better-stack-cloudtrail-logs.
  6. Optionally, specify a filter pattern to forward only a subset of logs from this log group.
  7. Press Start streaming.

Your re-routed logs should now appear in your new Better Stack source in Live tail.

Routing multiple log groups

You can repeat this process for as many log groups as you need. Each log group or set of log groups can be routed to a different Better Stack source with its own retention period.

You can also reuse a single Firehose stream for multiple log groups that should share the same Better Stack source — simply add a subscription filter to each log group pointing to the same stream.

Need help?

Please let us know at hello@betterstack.com. We're happy to help! 🙏

Previous article

Getting started with AWS

Next article

Better Stack Amazon CloudWatch log forwarding