Streaming responses are wonderful for users and miserable for observability. A single request is no longer one span with a start and an end — it’s a long-lived stream emitting tokens over seconds, crossing regions, buffers and network hops. Instrument it naively and your tracing bill grows faster than your traffic.
We wanted full visibility into every streamed token without turning our observability pipeline into the most expensive part of the stack. Here’s how we approached it.
Spans that follow the tokens
A streamed generation gets a single parent span, but the interesting signal lives inside it: time-to-first-token, inter-token latency, and where the request physically executed. Rather than emitting a span per token — which would be ruinous — we attach a compact histogram of inter-token gaps to the parent span and mark only the outliers as discrete events.
That keeps the cardinality bounded while still surfacing the two questions that matter: why was the first token slow, and did the stream stall partway through?
Sampling that respects the bill
The other half is sampling. We trace every slow or errored stream at full fidelity, and down-sample the fast, healthy ones aggressively. Because the parent span always carries the latency histogram, even sampled-out requests contribute to aggregate metrics — you lose the individual trace, not the statistics.
Across regions, we stitch traces together with a propagated context so a request that starts in Paris and fails over to Frankfurt still reads as one story. The net effect: end-to-end token-level insight, and an observability bill that scales with problems rather than with traffic.