# Ingesting traces

Better Stack ingests trace data using the [OpenTelemetry Protocol (OTLP)](https://betterstack.com/docs/logs/open-telemetry/), the open standard for telemetry data. This allows you to send distributed traces from your applications and services to gain deep visibility into your system's behavior.

For the easiest setup, we recommend using the [Better Stack collector](https://betterstack.com/docs/logs/collector/), which provides **zero-code auto-instrumentation** for Kubernetes and Docker.

[info]
#### New to distributed tracing?
Start with our [Introduction to Tracing](https://betterstack.com/docs/logs/tracing/) for an overview of how to explore your traces in Better Stack.
[/info]

[endpoint]
base_url = "/v1/traces"
method = "POST"

[[header]]
name = "Content-Type"
description = "Either `application/json` or `application/x-protobuf`."
required = true
type = "string"

[[header]]
name = "Authorization"
description = "Bearer `$SOURCE_TOKEN`"
required = true
type = "string"

[[body_param]]
name = "OTLP ExportTraceServiceRequest"
description = "A batch of spans structured according to the OpenTelemetry Protocol (OTLP) specification."
required = true
type = "object"
[/endpoint]

[responses]
[[response]]
status = 200
description = '''The trace data was successfully received.'''
body = '''{}'''

[[response]]
status = 401
description = '''You provided an invalid source token.'''
body = '''{"message":"Unauthorized"}'''
[/responses]

## cURL examples

[code-tabs]
```shell
[label Single span]
curl -X POST "https://$INGESTING_HOST/v1/traces" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $SOURCE_TOKEN" \
  --data-binary "{
    \"resourceSpans\": [
      {
        \"resource\": {
          \"attributes\": [
            {
              \"key\": \"service.name\",
              \"value\": { \"stringValue\": \"minimal-service\" }
            }
          ]
        },
        \"scopeSpans\": [
          {
            \"spans\": [
              {
                \"traceId\": \"$(openssl rand -hex 16)\",
                \"spanId\": \"$(openssl rand -hex 8)\",
                \"name\": \"Minimal span info\",
                \"startTimeUnixNano\": \"$(date +%s%N)\",
                \"endTimeUnixNano\": \"$(($(date +%s%N) + 1000000))\"
              }
            ]
          }
        ]
      }
    ]
  }"
```
```shell
[label Multiple spans]
TRACE_ID="$(openssl rand -hex 16)"
PARENT_SPAN_ID="$(openssl rand -hex 8)"

curl -X POST https://$INGESTING_HOST/v1/traces \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $SOURCE_TOKEN" \
  --data-binary "{
    \"resourceSpans\": [
      {
        \"resource\": {
          \"attributes\": [
            { \"key\": \"service.name\", \"value\": { \"stringValue\": \"my-api-service\" } }
          ]
        },
        \"scopeSpans\": [
          {
            \"scope\": { \"name\": \"my-instrumentation-library\" },
            \"spans\": [
              {
                \"traceId\": \"$TRACE_ID\",
                \"spanId\": \"$PARENT_SPAN_ID\",
                \"name\": \"GET /api/example\",
                \"kind\": 2,
                \"startTimeUnixNano\": \"$(date +%s%N)\",
                \"endTimeUnixNano\": \"$(($(date +%s%N) + 600000000))\",
                \"status\": { \"code\": 1 },
                \"attributes\": [
                  { \"key\": \"http.request.method\", \"value\": { \"stringValue\": \"GET\" } },
                  { \"key\": \"http.response.status_code\", \"value\": { \"intValue\": 200 } }
                ]
              },
              {
                \"traceId\": \"$TRACE_ID\",
                \"spanId\": \"$(openssl rand -hex 8)\",
                \"parentSpanId\": \"$PARENT_SPAN_ID\",
                \"name\": \"SELECT FROM examples\",
                \"kind\": 3,
                \"startTimeUnixNano\": \"$(($(date +%s%N) + 100000000))\",
                \"endTimeUnixNano\": \"$(($(date +%s%N) + 500000000))\",
                \"status\": { \"code\": 1 },
                \"attributes\": [
                  { \"key\": \"db.system\", \"value\": { \"stringValue\": \"mysql\" } },
                  { \"key\": \"db.statement\", \"value\": { \"stringValue\": \"SELECT * FROM examples WHERE id = ?\" } }
                ]
              }
            ]
          }
        ]
      }
    ]
  }"
```
```shell
[label Spans with events]
curl -X POST https://$INGESTING_HOST/v1/traces \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $SOURCE_TOKEN" \
  --data-binary "{
    \"resourceSpans\": [{
      \"resource\": { \"attributes\": [{ \"key\": \"service.name\", \"value\": { \"stringValue\": \"checkout-service\" } }] },
      \"scopeSpans\": [{
        \"spans\": [{
          \"traceId\": \"$(openssl rand -hex 16)\",
          \"spanId\": \"$(openssl rand -hex 8)\",
          \"name\": \"Process payment\",
          \"startTimeUnixNano\": \"$(date +%s%N)\",
          \"endTimeUnixNano\": \"$(($(date +%s%N) + 250000000))\",
          \"events\": [
            {
              \"timeUnixNano\": \"$(($(date +%s%N) + 50000000))\",
              \"name\": \"Payment validation started\"
            },
            {
              \"timeUnixNano\": \"$(($(date +%s%N) + 200000000))\",
              \"name\": \"Payment provider returned error\",
              \"attributes\": [
                { \"key\": \"error.code\", \"value\": { \"intValue\": 500 } },
                { \"key\": \"error.message\", \"value\": { \"stringValue\": \"Insufficient funds\" } }
              ]
            }
          ],
          \"status\": { \"code\": 2, \"message\": \"Payment failed\" }
        }]
      }]
    }]
  }"
```
[/code-tabs]

## OTLP data format

Traces are sent as a batch of spans. The root of the payload is a JSON object containing a `resourceSpans` array. Each item in this array represents spans from a specific resource (like a service or application).

```json
[label OTLP Payload Structure]
{
  "resourceSpans": [{
    "resource": {
      "attributes": [
        {"key": "service.name", "value": {"stringValue": "my-api-service"}}
      ]
    },
    "scopeSpans": [{
      "scope": {
        "name": "my-instrumentation-library"
      },
      "spans": [
        // ... An array of span objects ...
      ]
    }]
  }]
}
```

### Trace and span IDs

Each span represents a single operation within a trace.

| Field | Format | Length | Description |
|---|---|---|---|
| `traceId` | Hexadecimal string | 32 chars (16 bytes) | A unique identifier for the entire trace. All spans in a trace share the same `traceId`. |
| `spanId` | Hexadecimal string | 16 chars (8 bytes) | A unique identifier for the span. |
| `parentSpanId` | Hexadecimal string | 16 chars (8 bytes) | The `spanId` of the parent span. If this field is empty or missing, the span is a **root span**. |

[info]
Our ingestion service accepts IDs as either Base64-encoded binary (as per the OTLP protobuf specification) or direct hexadecimal strings (for non-conformant senders). For OTLP/JSON, use hex strings.
[/info]

### Correlating traces with logs

To get the most out of tracing, you can correlate your traces with logs. This allows you to jump from a specific span directly to the logs that were generated during that operation, providing full context for debugging.

To enable automatic correlation, ensure that your logs and traces include the same `span.trace_id` and `span.span_id` fields, that their timestamps fall within the span's duration, and are sent to the same Better Stack source.

When you send a log event via the [HTTP REST API for logs](https://betterstack.com/docs/logs/ingesting-data/http/logs/), you can include the trace context like this:

```json
[label Log event with trace context]
{
  "message": "User failed to log in: invalid password",
  "level": "error",
  "span": {
    "trace_id": "a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4",
    "span_id": "a1b2c3d4e5f6a1b2"
  }
}
```

[note]
#### Can't see your logs linked to your traces?

If your logs use different field names (like `trace_id` or `context.trace_id`), or if the [timestamp field](https://betterstack.com/docs/logs/ingesting-data/http/#sending-timestamps) is not correctly mapped to `.dt`, use a [VRL transformation](https://betterstack.com/docs/logs/using-logtail/transforming-ingested-data/logs-vrl/) on your source to rename them or map the correct timestamp upon ingest.

For example, using `.span.trace_id = del(.context.trace_id)` or `.dt = del(.timestamp)`.
[/note]

### Timestamps and duration

Timestamps are provided in nanoseconds since the Unix epoch.

| Field | Format | Example |
|---|---|---|
| `startTimeUnixNano` | Unix nanoseconds (string) | `"1738316288153703000"` |
| `endTimeUnixNano` | Unix nanoseconds (string) | `"1738316288208088000"` |

The `duration` of a span is calculated upon ingestion as `(endTimeUnixNano - startTimeUnixNano)`.

[warning]
Spans without an `endTimeUnixNano` are considered in-progress and are skipped during ingestion.
[/warning]

#### Core span details

*   `name`: A human-readable string for the operation (e.g., `"GET /api/users"`).
*   `kind`: An integer that specifies the type of operation.

| Integer | Kind | Description |
|---|---|---|
| `1` | `INTERNAL` | An internal operation within an application. |
| `2` | `SERVER` | A request handled by a server. |
| `3` | `CLIENT` | A request made by a client. |
| `4` | `PRODUCER` | A message sent to a messaging queue. |
| `5` | `CONSUMER` | A message received from a messaging queue. |

### Request limits

The maximum allowed size of a single request, which may contain many spans, is 10 MiB of compressed data.

There is **no limit to the number of requests** you can send.