Better Stack Azure Event Hub logging

Start logging in 5 minutes

Send logs to Better Stack from Azure Event Hub using Vector in Azure Container Instance.

Prerequisites

Make sure you have a Resource group and Event Hub created in your Azure account. Sign in to Azure CLI using az login.

1. Get Vector deployment

Copy and save the following configuration as vector-deploy.yaml:

Save configuration as vector-deploy.yaml
apiVersion: 2021-10-01
name: vector-betterstack
properties:
  containers:
  - name: vector
    properties:
      image: ghcr.io/vectordotdev/vector:latest-debian
      resources:
        requests:
          cpu: 1
          memoryInGb: 1
      command:
        - /bin/bash
        - -c
        - |
          cat > /etc/vector/vector.toml << 'EOF'
          data_dir = "/var/lib/vector"

          # Azure Event Hub source using Kafka protocol
          [sources.azure_eventhub]
          type = "kafka"
          bootstrap_servers = "EVENTHUB_NAMESPACE.servicebus.windows.net:9093"
          group_id = "vector-consumer-group"
          topics = ["EVENTHUB_NAME"]
          session_timeout_ms = 10000
          auto_offset_reset = "latest"

          [sources.azure_eventhub.sasl]
          enabled = true
          mechanism = "PLAIN"
          username = "$$ConnectionString"
          password = "EVENTHUB_CONNECTION_STRING"  # Your full Event Hub connection string starting in "Endpoint=..."

          [sources.azure_eventhub.librdkafka_options]
          "security.protocol" = "SASL_SSL"
          "sasl.mechanism" = "PLAIN"
          "sasl.username" = "$$ConnectionString"
          "sasl.password" = "EVENTHUB_CONNECTION_STRING"

          [transforms.rename_timestamp]
          type = "remap"
          inputs = ["azure_eventhub"]
          source = '.dt = .timestamp; del(.timestamp)'

          [sinks.better_stack]
          type = "http"
          method = "post"
          uri = "https://$INGESTING_HOST"
          inputs = ["rename_timestamp"]
          encoding.codec = "json"
          compression = "gzip"

          [sinks.better_stack.auth]
          strategy = "bearer"
          token = "$SOURCE_TOKEN"

          [sinks.better_stack.batch]
          max_bytes = 1048576
          timeout_secs = 1

          [sinks.better_stack.request]
          timeout_secs = 30
          retry_attempts = 3

          # API for debugging (optional)
          [api]
          enabled = true
          address = "0.0.0.0:8686"
          EOF
          echo "Starting Vector with Event Hub configuration..."
          vector --config /etc/vector/vector.toml
      environmentVariables:
        - name: VECTOR_LOG
          value: info
        - name: RUST_BACKTRACE
          value: full
      ports:
        - port: 8686
          protocol: TCP
  osType: Linux
  restartPolicy: Always
  ipAddress:
    type: Public
    ports:
      - port: 8686
        protocol: TCP
tags: null
type: Microsoft.ContainerInstance/containerGroups


2. Fill in your Azure Event Hub values

Fetch the following values using Azure CLI and fill them in the vector-deploy.yaml file:

Get EVENTHUB_NAMESPACE
az eventhubs namespace list --resource-group <YOUR_RESOURCE_GROUP> --query "[].name" -o tsv
Get EVENTHUB_NAME
az eventhubs eventhub list --resource-group <YOUR_RESOURCE_GROUP> \
  --namespace-name <YOUR_NAMESPACE> --query "[].name" -o tsv
Get EVENTHUB_CONNECTION_STRING
az eventhubs namespace authorization-rule keys list \
  --resource-group <YOUR_RESOURCE_GROUP> \
  --namespace-name <YOUR_NAMESPACE> \
  --name RootManageSharedAccessKey \
  --query primaryConnectionString -o tsv

3. Deploy Vector

Deploy Vector as Azure Container Instance using the following command:

Deploy Vector via Azure CLI
az container create \
  --location eastus \
  --resource-group <YOUR_RESOURCE_GROUP> \
  --file vector-deploy.yaml

Fill in your values

Replace location and resource groups parameters in the command above.

You should see your logs in Better Stack โ†’ Live tail.

Troubleshooting

Check the deployment status using the following commands:

Check deployment status
az container show --resource-group <YOUR_RESOURCE_GROUP> --name vector-betterstack --query instanceView.state
az container logs --resource-group <YOUR_RESOURCE_GROUP> --name vector-betterstack

Additional information

Want to learn more about Vector configuration and Azure Event Hub?

Check out the Vector Kafka source docs and Azure Event Hub Kafka protocol docs.