# Amazon Data Firehose logs & metrics

Send logs and CloudWatch metrics to Better Stack through [Amazon Data Firehose](https://aws.amazon.com/firehose/), formerly Kinesis Data Firehose.

Bring your own Firehose streams when the [Better Stack CloudFormation stack](https://betterstack.com/docs/logs/aws-cloudformation/) isn't an option for you, for example because of your account's IAM policy.

[info]
#### Already running the CloudFormation stack?
You already have both streams described here, `better-stack-logs` and `better-stack-metrics`. There is nothing to create.
[/info]

## Create a source

[Create a new AWS source](https://telemetry.betterstack.com/team/0/sources/new?platform=aws ";_blank") in Better Stack. One source can receive both logs and metrics.

## Create a Firehose stream

Logs and metrics use different Better Stack endpoints, and a Firehose stream has one destination. Create a stream for logs, a stream for metrics, or just the one you need.

In [AWS Console → Amazon Data Firehose → Firehose streams](https://console.aws.amazon.com/firehose/home#/streams), click **Create Firehose stream** and use these settings:

[code-tabs]
```plain
[label Logs stream configuration]
Source:               Direct PUT
Destination:          HTTP Endpoint
Firehose stream name: better-stack-logs
HTTP endpoint URL:    https://$INGESTING_HOST/aws-firehose
Access key:           $SOURCE_TOKEN
Content encoding:     GZIP
```
```plain
[label Metrics stream configuration]
Source:               Direct PUT
Destination:          HTTP Endpoint
Firehose stream name: better-stack-metrics
HTTP endpoint URL:    https://$INGESTING_HOST/aws-firehose/metrics
Access key:           $SOURCE_TOKEN
Content encoding:     GZIP
```
[/code-tabs]

Keep the remaining settings at their defaults:

- **Retry duration**: 300 seconds.
- **Buffer hints**: 5 MiB and 60 seconds. Better Stack accepts requests up to 10 MB.
- **Backup settings**: **Failed data only** with an S3 bucket of your choice. Firehose keeps the batches Better Stack couldn't accept there.
- **Service access**: **Create or update IAM role**.

[info]
#### Stream creation failed with "unable to assume role"?
The IAM role the console just created hasn't propagated yet. Wait a few seconds and press **Create Firehose stream** again.
[/info]

For the logs stream, run **Test with demo data** on the stream's detail page to check the connection. The demo records show up in [Live tail](https://telemetry.betterstack.com/team/0/tail ";_blank") within a few minutes. Skip it for the metrics stream. The demo records are not CloudWatch metrics and are discarded.

## Send logs

### From CloudWatch Logs

Forward a log group to your logs stream with a subscription filter:

1. Go to [CloudWatch → Log groups](https://console.aws.amazon.com/cloudwatch/home#logsV2:log-groups) and open the log group.
2. Switch to the **Subscription filters** tab and choose **Create → Create Amazon Data Firehose subscription filter**.
3. Select your logs Firehose stream as the destination.
4. Select an IAM role that CloudWatch Logs can assume to write to the stream. If you don't have one, create a role trusted by `logs.amazonaws.com` with the policy below, replacing the region and account ID.
5. Optionally enter a filter pattern to forward only matching events, then press **Start streaming**.

```json
[label IAM policy for the subscription filter role]
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["firehose:PutRecord", "firehose:PutRecordBatch"],
      "Resource": "arn:aws:firehose:<region>:<account-id>:deliverystream/better-stack-logs"
    }
  ]
}
```

Each log event arrives in Better Stack as a separate log with the log group and log stream names as context. Repeat the steps for every log group you want to forward. They can all share one Firehose stream.

To give log groups different retention periods, send them to separate sources through separate streams. If you use the CloudFormation stack, follow [Custom log group routing](https://betterstack.com/docs/logs/aws/custom-log-group-routing/).

### From other producers

Anything that writes JSON records to a Firehose stream works the same way, such as Fluent Bit's `kinesis_firehose` output on [AWS Fargate](https://betterstack.com/docs/logs/aws-fargate/) or AWS WAF logs.

## Stream CloudWatch metrics

A CloudWatch metric stream delivers every metric update in a region to your metrics Firehose stream within a few minutes.

1. [Create a metric stream](https://console.aws.amazon.com/cloudwatch/home#metric-streams:streamsList/create) in CloudWatch.
2. Under **Destination**, choose **Custom setup with Firehose** and select your metrics Firehose stream.
3. Expand **Change output format** and choose **JSON**.
4. Under **Metrics to be streamed**, keep **All metrics**, or choose **Select metrics** to include or exclude namespaces.
5. Name the stream and press **Create metric stream**.

![The Change output format section of the metric stream wizard, collapsed and defaulting to OpenTelemetry 1.0](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/85082e85-2f14-4fd6-36c9-e2d6efbb6200/lg2x =1184x817)

[warning]
#### Why JSON and not OpenTelemetry?
The console preselects the OpenTelemetry 1.0 format for custom Firehose setups. Better Stack ingests the JSON format only. Metrics streamed in an OpenTelemetry format are discarded. Switch an existing metric stream to JSON from its **Edit** page at any time.
[/warning]

Your metrics appear in [Dashboards](https://telemetry.betterstack.com/team/0/dashboards ";_blank") within a few minutes. To watch the Firehose stream itself, create a dashboard from the **AWS Firehose** template.

### Metrics in Better Stack

Metrics are converted to Better Stack naming. For example, `AWS/EC2/CPUUtilization` becomes `aws.ec2.cpu_utilization`, with its dimensions as tags such as `instance_id`. The CloudWatch statistics are available as `aws.ec2.cpu_utilization_min`, `_max`, `_sum` and `_count`.

### Costs and limits

- CloudWatch bills metric streams per metric update, and Firehose per gigabyte. Use **Select metrics** to limit the stream to the namespaces you need.
- AWS publishes metrics for global services such as Amazon CloudFront and Route 53 only in `us-east-1`. Create the metric stream there to collect them.
- A metric stream delivers to a single Firehose stream. To send the same metrics to Better Stack and to another destination, create a second metric stream.
- Metric streams are a per-region resource. Repeat the setup in every region you want to collect metrics from.

To trade latency for a lower bill, the CloudFormation stack can [poll your metrics through the CloudWatch API](https://betterstack.com/docs/logs/aws-cloudformation/#reducing-metric-costs-with-polling) instead.

## Deploy with CloudFormation

Prefer infrastructure as code? Each template creates the Firehose stream, its backup bucket and IAM roles, plus the producer - a subscription filter for one log group, or a metric stream with the output format set to JSON. Deploy them in each region you want to collect from.

[code-tabs]
```yaml
[label better-stack-logs.yaml]
AWSTemplateFormatVersion: '2010-09-09'
Description: Forward a CloudWatch log group to Better Stack through Amazon Data Firehose

Parameters:
  IngestingHost:
    Type: String
    Description: Ingesting host of your Better Stack source, for example s123456.eu-central-1a.betterstackdata.com
  SourceToken:
    Type: String
    NoEcho: true
    Description: Source token of your Better Stack source
  LogGroupName:
    Type: String
    Description: CloudWatch log group to forward, for example /aws/lambda/my-function

Resources:
  # Firehose requires a bucket for records it could not deliver
  BackupBucket:
    Type: AWS::S3::Bucket

  FirehoseRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: firehose.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: BackupBucketWrite
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - s3:PutObject
                  - s3:GetBucketLocation
                  - s3:ListBucket
                Resource:
                  - !GetAtt BackupBucket.Arn
                  - !Sub '${BackupBucket.Arn}/*'

  LogsFirehose:
    Type: AWS::KinesisFirehose::DeliveryStream
    Properties:
      DeliveryStreamName: better-stack-logs
      DeliveryStreamType: DirectPut
      HttpEndpointDestinationConfiguration:
        EndpointConfiguration:
          Url: !Sub 'https://${IngestingHost}/aws-firehose'
          AccessKey: !Ref SourceToken
        RequestConfiguration:
          ContentEncoding: GZIP
        BufferingHints:
          SizeInMBs: 5
          IntervalInSeconds: 60
        RoleARN: !GetAtt FirehoseRole.Arn
        S3Configuration:
          BucketARN: !GetAtt BackupBucket.Arn
          RoleARN: !GetAtt FirehoseRole.Arn

  SubscriptionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: logs.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: FirehoseWrite
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - firehose:PutRecord
                  - firehose:PutRecordBatch
                Resource: !GetAtt LogsFirehose.Arn

  # Add one SubscriptionFilter per log group you want to forward
  SubscriptionFilter:
    Type: AWS::Logs::SubscriptionFilter
    Properties:
      LogGroupName: !Ref LogGroupName
      DestinationArn: !GetAtt LogsFirehose.Arn
      RoleArn: !GetAtt SubscriptionRole.Arn
      FilterPattern: ''
```
```yaml
[label better-stack-metrics.yaml]
AWSTemplateFormatVersion: '2010-09-09'
Description: Stream CloudWatch metrics to Better Stack through Amazon Data Firehose

Parameters:
  IngestingHost:
    Type: String
    Description: Ingesting host of your Better Stack source, for example s123456.eu-central-1a.betterstackdata.com
  SourceToken:
    Type: String
    NoEcho: true
    Description: Source token of your Better Stack source

Resources:
  # Firehose requires a bucket for records it could not deliver
  BackupBucket:
    Type: AWS::S3::Bucket

  FirehoseRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: firehose.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: BackupBucketWrite
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - s3:PutObject
                  - s3:GetBucketLocation
                  - s3:ListBucket
                Resource:
                  - !GetAtt BackupBucket.Arn
                  - !Sub '${BackupBucket.Arn}/*'

  MetricsFirehose:
    Type: AWS::KinesisFirehose::DeliveryStream
    Properties:
      DeliveryStreamName: better-stack-metrics
      DeliveryStreamType: DirectPut
      HttpEndpointDestinationConfiguration:
        EndpointConfiguration:
          Url: !Sub 'https://${IngestingHost}/aws-firehose/metrics'
          AccessKey: !Ref SourceToken
        RequestConfiguration:
          ContentEncoding: GZIP
        BufferingHints:
          SizeInMBs: 5
          IntervalInSeconds: 60
        RoleARN: !GetAtt FirehoseRole.Arn
        S3Configuration:
          BucketARN: !GetAtt BackupBucket.Arn
          RoleARN: !GetAtt FirehoseRole.Arn

  MetricStreamRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: streams.metrics.cloudwatch.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: FirehoseWrite
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - firehose:PutRecord
                  - firehose:PutRecordBatch
                Resource: !GetAtt MetricsFirehose.Arn

  MetricStream:
    Type: AWS::CloudWatch::MetricStream
    Properties:
      Name: better-stack-metric-stream
      FirehoseArn: !GetAtt MetricsFirehose.Arn
      RoleArn: !GetAtt MetricStreamRole.Arn
      OutputFormat: json
```
[/code-tabs]

Deploy with the AWS CLI:

[code-tabs]
```sh
[label Deploy the logs stack]
aws cloudformation deploy \
  --stack-name better-stack-logs \
  --template-file better-stack-logs.yaml \
  --capabilities CAPABILITY_IAM \
  --parameter-overrides \
    IngestingHost=$INGESTING_HOST \
    SourceToken=$SOURCE_TOKEN \
    LogGroupName=/aws/lambda/my-function
```
```sh
[label Deploy the metrics stack]
aws cloudformation deploy \
  --stack-name better-stack-metrics \
  --template-file better-stack-metrics.yaml \
  --capabilities CAPABILITY_IAM \
  --parameter-overrides \
    IngestingHost=$INGESTING_HOST \
    SourceToken=$SOURCE_TOKEN
```
[/code-tabs]

## Troubleshooting

[info]
#### Not seeing any data?
Open the stream's **Monitoring** tab for the HTTP endpoint delivery success rate and the **Destination error logs** tab for rejected deliveries. Batches that failed for longer than the retry duration end up in the backup bucket under the error output prefix. For metrics, also check that the metric stream's output format is JSON.
[/info]

[info]
#### Seeing InvalidResponseFromDestination in the destination error logs?
Better Stack rejected the request. The usual cause is an access key that doesn't match your source token. A paused source or an exceeded quota are the other reasons. Fix the cause and Firehose retries the pending batches on its own.
[/info]

## Pause or remove the integration

Firehose streams can't be paused. Stop the metric stream in [CloudWatch → Metrics → Streams](https://console.aws.amazon.com/cloudwatch/home#metric-streams:streamsList), or delete the subscription filter on a log group. Delivery resumes when you start it again.

To remove the integration, delete the metric stream and the subscription filters first, then the Firehose streams, the IAM roles, and the backup bucket after emptying it.

## Need help?

Please let us know at hello@betterstack.com.  
We're happy to help! 🙏