Amazon Data Firehose logs & metrics

Send logs and CloudWatch metrics to Better Stack through Amazon Data Firehose, formerly Kinesis Data Firehose.

Bring your own Firehose streams when the Better Stack CloudFormation stack isn't an option for you, for example because of your account's IAM policy.

Create a source

Create a new AWS source in Better Stack. One source can receive both logs and metrics.

Create a Firehose stream

Logs and metrics use different Better Stack endpoints, and a Firehose stream has one destination. Create a stream for logs, a stream for metrics, or just the one you need.

In AWS Console → Amazon Data Firehose → Firehose streams, click Create Firehose stream and use these settings:

Source:               Direct PUT
Destination:          HTTP Endpoint
Firehose stream name: better-stack-logs
HTTP endpoint URL:    https://$INGESTING_HOST/aws-firehose
Access key:           $SOURCE_TOKEN
Content encoding:     GZIP

Source:               Direct PUT
Destination:          HTTP Endpoint
Firehose stream name: better-stack-metrics
HTTP endpoint URL:    https://$INGESTING_HOST/aws-firehose/metrics
Access key:           $SOURCE_TOKEN
Content encoding:     GZIP

Keep the remaining settings at their defaults:

• Retry duration: 300 seconds.
• Buffer hints: 5 MiB and 60 seconds. Better Stack accepts requests up to 10 MB.
• Backup settings: Failed data only with an S3 bucket of your choice. Firehose keeps the batches Better Stack couldn't accept there.
• Service access: Create or update IAM role.

For the logs stream, run Test with demo data on the stream's detail page to check the connection. The demo records show up in Live tail within a few minutes. Skip it for the metrics stream. The demo records are not CloudWatch metrics and are discarded.

Send logs

From CloudWatch Logs

Forward a log group to your logs stream with a subscription filter:

1. Go to CloudWatch → Log groups and open the log group.
2. Switch to the Subscription filters tab and choose Create → Create Amazon Data Firehose subscription filter.
3. Select your logs Firehose stream as the destination.
4. Select an IAM role that CloudWatch Logs can assume to write to the stream. If you don't have one, create a role trusted by logs.amazonaws.com with the policy below, replacing the region and account ID.
5. Optionally enter a filter pattern to forward only matching events, then press Start streaming.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["firehose:PutRecord", "firehose:PutRecordBatch"],
      "Resource": "arn:aws:firehose:<region>:<account-id>:deliverystream/better-stack-logs"
    }
  ]
}

Each log event arrives in Better Stack as a separate log with the log group and log stream names as context. Repeat the steps for every log group you want to forward. They can all share one Firehose stream.

To give log groups different retention periods, send them to separate sources through separate streams. If you use the CloudFormation stack, follow Custom log group routing.

From other producers

Anything that writes JSON records to a Firehose stream works the same way, such as Fluent Bit's kinesis_firehose output on AWS Fargate or AWS WAF logs.

Stream CloudWatch metrics

A CloudWatch metric stream delivers every metric update in a region to your metrics Firehose stream within a few minutes.

1. Create a metric stream in CloudWatch.
2. Under Destination, choose Custom setup with Firehose and select your metrics Firehose stream.
3. Expand Change output format and choose JSON.
4. Under Metrics to be streamed, keep All metrics, or choose Select metrics to include or exclude namespaces.
5. Name the stream and press Create metric stream.

Your metrics appear in Dashboards within a few minutes. To watch the Firehose stream itself, create a dashboard from the AWS Firehose template.

Metrics in Better Stack

Metrics are converted to Better Stack naming. For example, AWS/EC2/CPUUtilization becomes aws.ec2.cpu_utilization, with its dimensions as tags such as instance_id. The CloudWatch statistics are available as aws.ec2.cpu_utilization_min, _max, _sum and _count.

Costs and limits

• CloudWatch bills metric streams per metric update, and Firehose per gigabyte. Use Select metrics to limit the stream to the namespaces you need.
• AWS publishes metrics for global services such as Amazon CloudFront and Route 53 only in us-east-1. Create the metric stream there to collect them.
• A metric stream delivers to a single Firehose stream. To send the same metrics to Better Stack and to another destination, create a second metric stream.
• Metric streams are a per-region resource. Repeat the setup in every region you want to collect metrics from.

To trade latency for a lower bill, the CloudFormation stack can poll your metrics through the CloudWatch API instead.

Deploy with CloudFormation

Prefer infrastructure as code? Each template creates the Firehose stream, its backup bucket and IAM roles, plus the producer - a subscription filter for one log group, or a metric stream with the output format set to JSON. Deploy them in each region you want to collect from.

AWSTemplateFormatVersion: '2010-09-09'
Description: Forward a CloudWatch log group to Better Stack through Amazon Data Firehose

Parameters:
  IngestingHost:
    Type: String
    Description: Ingesting host of your Better Stack source, for example s123456.eu-central-1a.betterstackdata.com
  SourceToken:
    Type: String
    NoEcho: true
    Description: Source token of your Better Stack source
  LogGroupName:
    Type: String
    Description: CloudWatch log group to forward, for example /aws/lambda/my-function

Resources:
  # Firehose requires a bucket for records it could not deliver
  BackupBucket:
    Type: AWS::S3::Bucket

  FirehoseRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: firehose.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: BackupBucketWrite
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - s3:PutObject
                  - s3:GetBucketLocation
                  - s3:ListBucket
                Resource:
                  - !GetAtt BackupBucket.Arn
                  - !Sub '${BackupBucket.Arn}/*'

  LogsFirehose:
    Type: AWS::KinesisFirehose::DeliveryStream
    Properties:
      DeliveryStreamName: better-stack-logs
      DeliveryStreamType: DirectPut
      HttpEndpointDestinationConfiguration:
        EndpointConfiguration:
          Url: !Sub 'https://${IngestingHost}/aws-firehose'
          AccessKey: !Ref SourceToken
        RequestConfiguration:
          ContentEncoding: GZIP
        BufferingHints:
          SizeInMBs: 5
          IntervalInSeconds: 60
        RoleARN: !GetAtt FirehoseRole.Arn
        S3Configuration:
          BucketARN: !GetAtt BackupBucket.Arn
          RoleARN: !GetAtt FirehoseRole.Arn

  SubscriptionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: logs.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: FirehoseWrite
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - firehose:PutRecord
                  - firehose:PutRecordBatch
                Resource: !GetAtt LogsFirehose.Arn

  # Add one SubscriptionFilter per log group you want to forward
  SubscriptionFilter:
    Type: AWS::Logs::SubscriptionFilter
    Properties:
      LogGroupName: !Ref LogGroupName
      DestinationArn: !GetAtt LogsFirehose.Arn
      RoleArn: !GetAtt SubscriptionRole.Arn
      FilterPattern: ''

AWSTemplateFormatVersion: '2010-09-09'
Description: Stream CloudWatch metrics to Better Stack through Amazon Data Firehose

Parameters:
  IngestingHost:
    Type: String
    Description: Ingesting host of your Better Stack source, for example s123456.eu-central-1a.betterstackdata.com
  SourceToken:
    Type: String
    NoEcho: true
    Description: Source token of your Better Stack source

Resources:
  # Firehose requires a bucket for records it could not deliver
  BackupBucket:
    Type: AWS::S3::Bucket

  FirehoseRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: firehose.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: BackupBucketWrite
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - s3:PutObject
                  - s3:GetBucketLocation
                  - s3:ListBucket
                Resource:
                  - !GetAtt BackupBucket.Arn
                  - !Sub '${BackupBucket.Arn}/*'

  MetricsFirehose:
    Type: AWS::KinesisFirehose::DeliveryStream
    Properties:
      DeliveryStreamName: better-stack-metrics
      DeliveryStreamType: DirectPut
      HttpEndpointDestinationConfiguration:
        EndpointConfiguration:
          Url: !Sub 'https://${IngestingHost}/aws-firehose/metrics'
          AccessKey: !Ref SourceToken
        RequestConfiguration:
          ContentEncoding: GZIP
        BufferingHints:
          SizeInMBs: 5
          IntervalInSeconds: 60
        RoleARN: !GetAtt FirehoseRole.Arn
        S3Configuration:
          BucketARN: !GetAtt BackupBucket.Arn
          RoleARN: !GetAtt FirehoseRole.Arn

  MetricStreamRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: streams.metrics.cloudwatch.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: FirehoseWrite
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - firehose:PutRecord
                  - firehose:PutRecordBatch
                Resource: !GetAtt MetricsFirehose.Arn

  MetricStream:
    Type: AWS::CloudWatch::MetricStream
    Properties:
      Name: better-stack-metric-stream
      FirehoseArn: !GetAtt MetricsFirehose.Arn
      RoleArn: !GetAtt MetricStreamRole.Arn
      OutputFormat: json

Deploy with the AWS CLI:

aws cloudformation deploy \
  --stack-name better-stack-logs \
  --template-file better-stack-logs.yaml \
  --capabilities CAPABILITY_IAM \
  --parameter-overrides \
    IngestingHost=$INGESTING_HOST \
    SourceToken=$SOURCE_TOKEN \
    LogGroupName=/aws/lambda/my-function

aws cloudformation deploy \
  --stack-name better-stack-metrics \
  --template-file better-stack-metrics.yaml \
  --capabilities CAPABILITY_IAM \
  --parameter-overrides \
    IngestingHost=$INGESTING_HOST \
    SourceToken=$SOURCE_TOKEN

Troubleshooting

Pause or remove the integration

Firehose streams can't be paused. Stop the metric stream in CloudWatch → Metrics → Streams, or delete the subscription filter on a log group. Delivery resumes when you start it again.

To remove the integration, delete the metric stream and the subscription filters first, then the Firehose streams, the IAM roles, and the backup bucket after emptying it.

Need help?

Please let us know at hello@betterstack.com.
We're happy to help! 🙏