Explore documentation
  1. Welcome to Telemetry 👋
  2. Quick start guide
  3. Wide events vs. metrics
  4. Ingesting data
    1. Better Stack collector
    2. Vector
    3. OpenTelemetry
    4. Ingesting via HTTP API
      1. Ingesting logs
      2. Ingesting metrics
      3. Ingesting traces
    5. Ingesting metrics
    6. Ingesting logs
    7. Migrating to Better Stack
  5. Using the product
    1. AI SRE ↗
    2. MCP server ↗
    3. Transforming ingested data
    4. Querying data
    5. Tracing
    6. Alerts
  6. Telemetry API
    1. Getting an API token
    2. Sources API
    3. Source groups API
    4. Metrics API
    5. Dashboards API
    6. Connections API
    7. Collectors API
    8. Query API ↗
  7. Alternative to
    1. Lightstep

Ingesting traces

Better Stack ingests trace data using the OpenTelemetry Protocol (OTLP), the open standard for telemetry data. This allows you to send distributed traces from your applications and services to gain deep visibility into your system's behavior.

For the easiest setup, we recommend using the Better Stack collector, which provides zero-code auto-instrumentation for Kubernetes and Docker.

New to distributed tracing?

Start with our Introduction to Tracing for an overview of how to explore your traces in Better Stack.

POST /v1/traces

Headers

Content-Type
required string

Either application/json or application/x-protobuf.

Authorization
required string

Bearer $SOURCE_TOKEN

Body parameters

OTLP ExportTraceServiceRequest
required object

A batch of spans structured according to the OpenTelemetry Protocol (OTLP) specification.

200

The trace data was successfully received.

Response body

{}
401

You provided an invalid source token.

Response body

{
  "message": "Unauthorized"
}

cURL examples

Single span Multiple spans Spans with events 
curl -X POST "https://$INGESTING_HOST/v1/traces" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $SOURCE_TOKEN" \
  --data-binary "{
    \"resourceSpans\": [
      {
        \"resource\": {
          \"attributes\": [
            {
              \"key\": \"service.name\",
              \"value\": { \"stringValue\": \"minimal-service\" }
            }
          ]
        },
        \"scopeSpans\": [
          {
            \"spans\": [
              {
                \"traceId\": \"$(openssl rand -hex 16)\",
                \"spanId\": \"$(openssl rand -hex 8)\",
                \"name\": \"Minimal span info\",
                \"startTimeUnixNano\": \"$(date +%s%N)\",
                \"endTimeUnixNano\": \"$(($(date +%s%N) + 1000000))\"
              }
            ]
          }
        ]
      }
    ]
  }"
TRACE_ID="$(openssl rand -hex 16)"
PARENT_SPAN_ID="$(openssl rand -hex 8)"

curl -X POST https://$INGESTING_HOST/v1/traces \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $SOURCE_TOKEN" \
  --data-binary "{
    \"resourceSpans\": [
      {
        \"resource\": {
          \"attributes\": [
            { \"key\": \"service.name\", \"value\": { \"stringValue\": \"my-api-service\" } }
          ]
        },
        \"scopeSpans\": [
          {
            \"scope\": { \"name\": \"my-instrumentation-library\" },
            \"spans\": [
              {
                \"traceId\": \"$TRACE_ID\",
                \"spanId\": \"$PARENT_SPAN_ID\",
                \"name\": \"GET /api/example\",
                \"kind\": 2,
                \"startTimeUnixNano\": \"$(date +%s%N)\",
                \"endTimeUnixNano\": \"$(($(date +%s%N) + 600000000))\",
                \"status\": { \"code\": 1 },
                \"attributes\": [
                  { \"key\": \"http.request.method\", \"value\": { \"stringValue\": \"GET\" } },
                  { \"key\": \"http.response.status_code\", \"value\": { \"intValue\": 200 } }
                ]
              },
              {
                \"traceId\": \"$TRACE_ID\",
                \"spanId\": \"$(openssl rand -hex 8)\",
                \"parentSpanId\": \"$PARENT_SPAN_ID\",
                \"name\": \"SELECT FROM examples\",
                \"kind\": 3,
                \"startTimeUnixNano\": \"$(($(date +%s%N) + 100000000))\",
                \"endTimeUnixNano\": \"$(($(date +%s%N) + 500000000))\",
                \"status\": { \"code\": 1 },
                \"attributes\": [
                  { \"key\": \"db.system\", \"value\": { \"stringValue\": \"mysql\" } },
                  { \"key\": \"db.statement\", \"value\": { \"stringValue\": \"SELECT * FROM examples WHERE id = ?\" } }
                ]
              }
            ]
          }
        ]
      }
    ]
  }"
curl -X POST https://$INGESTING_HOST/v1/traces \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $SOURCE_TOKEN" \
  --data-binary "{
    \"resourceSpans\": [{
      \"resource\": { \"attributes\": [{ \"key\": \"service.name\", \"value\": { \"stringValue\": \"checkout-service\" } }] },
      \"scopeSpans\": [{
        \"spans\": [{
          \"traceId\": \"$(openssl rand -hex 16)\",
          \"spanId\": \"$(openssl rand -hex 8)\",
          \"name\": \"Process payment\",
          \"startTimeUnixNano\": \"$(date +%s%N)\",
          \"endTimeUnixNano\": \"$(($(date +%s%N) + 250000000))\",
          \"events\": [
            {
              \"timeUnixNano\": \"$(($(date +%s%N) + 50000000))\",
              \"name\": \"Payment validation started\"
            },
            {
              \"timeUnixNano\": \"$(($(date +%s%N) + 200000000))\",
              \"name\": \"Payment provider returned error\",
              \"attributes\": [
                { \"key\": \"error.code\", \"value\": { \"intValue\": 500 } },
                { \"key\": \"error.message\", \"value\": { \"stringValue\": \"Insufficient funds\" } }
              ]
            }
          ],
          \"status\": { \"code\": 2, \"message\": \"Payment failed\" }
        }]
      }]
    }]
  }"


OTLP data format

Traces are sent as a batch of spans. The root of the payload is a JSON object containing a resourceSpans array. Each item in this array represents spans from a specific resource (like a service or application).

OTLP Payload Structure
{
  "resourceSpans": [{
    "resource": {
      "attributes": [
        {"key": "service.name", "value": {"stringValue": "my-api-service"}}
      ]
    },
    "scopeSpans": [{
      "scope": {
        "name": "my-instrumentation-library"
      },
      "spans": [
        // ... An array of span objects ...
      ]
    }]
  }]
}

Trace and span IDs

Each span represents a single operation within a trace.

Field Format Length Description
traceId Hexadecimal string 32 chars (16 bytes) A unique identifier for the entire trace. All spans in a trace share the same traceId.
spanId Hexadecimal string 16 chars (8 bytes) A unique identifier for the span.
parentSpanId Hexadecimal string 16 chars (8 bytes) The spanId of the parent span. If this field is empty or missing, the span is a root span.

Our ingestion service accepts IDs as either Base64-encoded binary (as per the OTLP protobuf specification) or direct hexadecimal strings (for non-conformant senders). For OTLP/JSON, use hex strings.

Correlating traces with logs

To get the most out of tracing, you can correlate your traces with logs. This allows you to jump from a specific span directly to the logs that were generated during that operation, providing full context for debugging.

To enable automatic correlation, ensure that your logs and traces include the same span.trace_id and span.span_id fields, and are sent to the same Better Stack source.

When you send a log event via the HTTP REST API for logs, you can include the trace context like this:

Log event with trace context
{
  "message": "User failed to log in: invalid password",
  "level": "error",
  "span": {
    "trace_id": "a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4",
    "span_id": "a1b2c3d4e5f6a1b2"
  }
}

Timestamps and duration

Timestamps are provided in nanoseconds since the Unix epoch.

Field Format Example
startTimeUnixNano Unix nanoseconds (string) "1738316288153703000"
endTimeUnixNano Unix nanoseconds (string) "1738316288208088000"

The duration of a span is calculated upon ingestion as (endTimeUnixNano - startTimeUnixNano).

Spans without an endTimeUnixNano are considered in-progress and are skipped during ingestion.

Core span details

  • name: A human-readable string for the operation (e.g., "GET /api/users").
  • kind: An integer that specifies the type of operation.
Integer Kind Description
1 INTERNAL An internal operation within an application.
2 SERVER A request handled by a server.
3 CLIENT A request made by a client.
4 PRODUCER A message sent to a messaging queue.
5 CONSUMER A message received from a messaging queue.

Request limits

The maximum allowed size of a single request, which may contain many spans, is 10 MiB of compressed data.

There is no limit to the number of requests you can send.

Previous article

Ingesting metrics

Next article

Better Stack Prometheus metrics scraping