GLASS BOX.
trace agents with opentelemetry / from a black box to a glass house
your agents are a black box. tonight we make it glass.
THE ANSWER WAS WRONG. WHICH AGENT BROKE?
Your planner called a researcher called a coder, and the final answer came back wrong. The logs show a few scattered lines. You cannot tell where in the trajectory it stopped making progress.
Agents fail mid-run, and the failure point is invisible to anything that only looks at the final answer.
[F1]
BLACK BOX. GLASS BOX. GLASS HOUSE.
Logs tell you what happened. A trace tells you where, and how fast. We turn the box to glass, one span at a time.
[F2] [BOARD]
"IT TOOK 8 SECONDS" IS NOT THE SAME AS "WHERE".
[F2]
AGENTS FAIL MID-RUN, NOT AT THE END.
An agent spends compute across tokens, tool calls, retries, and code. When a run fails, the final answer shows the endpoint, not the point where the trajectory stopped making recoverable progress. The failure modes map onto signals that live in the spans.
[F1]
ONE AGENT IS HARD. A SOCIETY IS THE BLIND SPOT.
Multi-agent systems fail in systematic, nameable ways.
A single agent you can babysit. But when a planner delegates to a researcher that delegates to a coder, the failure hides in the handoffs, the place your logs go quiet. The blind spot is not inside an agent. It is between them. [F3]
THE BUG IS IN THE LOGS. SO ARE TEN THOUSAND OTHER LINES.
The handful of causally-decisive events, the route taken, the memory written, the tool that failed, are buried in unstructured log spew. The win comes from compiling the run into a structured trace.
OTel's span model is that structure, off the shelf.
Parent and child nesting, typed attributes, an explicit status: the run stops being a wall of text and becomes a tree you can read. [F4]
YOU ARE NOT EARLY. YOU ARE ON STANDARD.
The 2026 agent-framework landscape already runs on traces:
Standardize on OTel and the gen_ai conventions: your traces stay portable across backends instead of locked to one vendor's UI. [F5]
A TRACE IS THE PATH. A SPAN IS ONE STEP.
[OTEL-TRACES]
NINE FIELDS, AND THEY ALL EARN THEIR PLACE.
trace id, span id, flags, state (immutable)[OTEL-TRACES]
CONTEXT IS HOW ONE TRACE SPANS MANY STEPS.
A span is not an island. The thing that links a tool call to the model call that triggered it is the propagated context: the same trace id carried from parent to child.
a child's parent_id equals its parent's span_id
that nesting IS the waterfall
Hold onto this. When two agents talk, the only thing that keeps it one trace is carrying that context across the boundary. That is the whole game in Act 4. [OTEL-TRACES]
SPANS HAVE TO GO SOMEWHERE.
You do not read spans in your code. A small pipeline ships them out to a place you can query.
Tracer Provider (one per app) → Tracer → creates spans → Exporter
→ stdout | the OpenTelemetry Collector | a backend
[OTEL-TRACES]
THE GEN_AI CONVENTIONS. experimental
Standard attribute keys so every tool emits comparable traces. Stability is "Development": cite them, but treat them as a moving target, not a contract.
gen_ai.provider.name
gen_ai.operation.name
gen_ai.request.model
gen_ai.request.temperature
gen_ai.request.max_tokens
gen_ai.request.top_p
gen_ai.response.model
gen_ai.response.id
gen_ai.response.finish_reasons
gen_ai.usage.input_tokens
gen_ai.usage.output_tokens
// superseded: gen_ai.system,
// prompt_tokens, completion_tokens
[OTEL-GENAI]
FOUR MOVES AND THE CALL IS VISIBLE.
gen_ai.* attributes on the model span.Status = Error on the span when the step fails.The runnable code lives in the lab, not on this slide. How the prompt and completion text itself is captured is still evolving in the conventions, so we instrument the metadata above and label any captured content as illustrative. [OTEL-TRACES] [OTEL-GENAI] · FACTS Open-items #1
YOU CAN NOW DESCRIBE ANY SINGLE CALL.
A trace, a span, the fields, the context that links them, the gen_ai keys, and four moves to instrument a call. That is the whole primer.
Next: point it at a real agent and read the waterfall. Then the hard part: what happens when the agent is not alone. [OTEL-TRACES]
A REAL AGENT RUN, AS A WATERFALL.
One root span for the agent turn, a child span for each step: prompt assembly, the tool call, the LLM call, the response. The nesting is parent_id; the bar length is start to end. [OTEL-TRACES]
THE LONGEST BAR IS YOUR LATENCY.
"Why is this agent slow?" stops being a guess.
start and end on every span show exactly which step ate the time, usually a tool call or the model. You optimize the span that actually hurts, not the one you assumed. [OTEL-TRACES]
EVERY MODEL SPAN CARRIES ITS BILL.
The token cost has a cause you can point at.
gen_ai.usage.input_tokens and gen_ai.usage.output_tokens per model span turn a monthly surprise into a per-step number. The runaway prompt is right there on the trace. [OTEL-GENAI]
STATUS = ERROR LANDS YOU ON THE FAILING STEP.
Not "the run failed." This span failed.
Span Status is Unset, Ok, or Error. The errored span is flagged in the waterfall, so a flaky tool or a bad parse surfaces as one red bar instead of a mystery. [OTEL-TRACES]
WHY DID THIS RUN DIFFER FROM THE LAST?
The trace remembers what the model forgot.
gen_ai.request.model, the request params, and gen_ai.response.finish_reasons are captured on the span, so two diverging runs become a diff you can read instead of a shrug. [OTEL-GENAI]
Four reads, one agent. But your agents are not alone. What happens when one agent calls another? [OTEL-TRACES]
ONE AGENT WAS THE EASY PART.
Real systems are not one agent. They are a planner that calls a researcher that calls a coder. The interesting failures live in the handoffs, in the space between agents.
Black box to glass box was one agent. Now the glass HOUSE: a whole society of agents, every conversation observable.
[A2A]
A2A: AN OPEN STANDARD FOR AGENTS THAT CALL AGENTS.
Not an agent calling a tool. One independent agent system calling another, over a real protocol. [A2A] v1.0.0
[A2A]
THE A2A HANDSHAKE.
SendMessageRequest carrying a Message (role + Parts).Task (or a direct Message for a simple turn).[A2A]
EVERY TURN OF THE HANDSHAKE IS A SPAN.
An A2A call is, under the hood, an HTTP request and a response. Wrap each turn in a span and the conversation becomes readable.
[A2A] [OTEL-TRACES]
HOW DOES ONE TRACE CROSS A NETWORK CALL?
In Act 2 context linked spans inside one process. Across agents, the same idea rides the wire: the carrier is the W3C Trace Context standard, two HTTP headers, traceparent and tracestate.
If the context crosses the boundary, two agents are one trace. If it does not, you have two black boxes again.
[W3C-TRACECTX]
READ A TRACEPARENT.
00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
[W3C-TRACECTX] · tracestate carries vendor pairs alongside
TWO AGENTS, ONE TRACE.
traceparent into the outgoing A2A request→
Same trace-id. B's spans point at A's parent-id. The two agents' work joins into one waterfall. [W3C-TRACECTX] [OTEL-TRACES]
THE MULTI-AGENT WATERFALL.
One trace whose root is the originating turn, with nested spans crossing into each downstream agent: A calls B calls C, every agent's own LLM and tool spans beneath it. Propagation working end to end. [W3C-TRACECTX] [OTEL-TRACES]
FOUR WAYS THE HOUSE GOES DARK.
traceparent, so its work starts a new trace and the trail ends at the handoff.parent-id points at a parent that never arrived, floating unattached.[W3C-TRACECTX] [OTEL-TRACES] [OTEL-GENAI] · read one for real in Act 5
A WHOLE SOCIETY OF AGENTS, AND YOU CAN SEE INSIDE IT.
One agent was the glass box. Every conversation between them is the glass house.
The mechanic is not magic: shared trace-id, propagated context, spans that nest across the network. Next we read a real one and find the broken handoff. [W3C-TRACECTX]
ONE TRACE, THREE AGENTS, NO SECRETS.
A planner calls a researcher calls a coder. One trace, the planner's turn at the root, each downstream agent's spans nested beneath the call that spawned them. Now we read it and find the bug. [W3C-TRACECTX] [OTEL-TRACES]
THE BROKEN HANDOFF.
A says it finished. B never actually ran.
Agent A handed off to Agent B, but B errored or never started. Without the trace you only see A's optimistic output and a wrong final answer with no obvious cause. [OTEL-TRACES]
the tell: a missing child span at the boundary, or one with Status = Error right where the handoff should be
THE RUNAWAY LOOP.
Two agents, each waiting for the other to be done.
Agents calling each other in a cycle burn tokens and wall-clock with nothing to show. In the trace it is unmistakable: a deep, repeating span chain that does not terminate. [F1] [OTEL-TRACES]
planner → researcher → planner → researcher → ...
THE COST BLOWUP.
The bill is a sum across agents you could not see.
Read gen_ai.usage.input_tokens and gen_ai.usage.output_tokens on every model span and roll it up per agent. Suddenly the researcher that quietly re-queries five times is the line item, not a mystery on the invoice. [OTEL-GENAI]
TWO MORE TELLS, IN THE SHAPE.
traceparent, so its work started a new trace.parent-id points at a parent that never exported, floating unattached.The method: start at the root, follow the nesting, and look for the span that is missing, errored, repeating, or expensive. [W3C-TRACECTX] [OTEL-TRACES]
NOW YOU MAKE THE BOX GLASS.
The hands-on hour. Three labs, copy-paste runnable, that walk from one agent to a whole conversation. Pair up and start.
[BOARD]
INSTRUMENT A TINY AGENT. SEE THE WATERFALL.
A self-contained program: a one-agent run, OpenTelemetry added, a real span tree printed to your console. Read it in two minutes, then make it yours.
[BOARD]
WATCH A CONVERSATION BECOME A TRACE.
Agent A calls Agent B. The traceparent crosses the boundary, B's spans nest under A's, and the two agents resolve into one waterfall in front of you.
[BOARD] [W3C-TRACECTX]
CLICK A SPAN. READ ITS ATTRIBUTES.
An interactive trace explorer. Click into any span to see its name, timing, status, and the gen_ai attributes on it. This is what a backend gives you, in miniature, in your browser.
[BOARD] [OTEL-TRACES]
FIVE MOVES AND YOUR STACK IS A GLASS HOUSE.
gen_ai.* attributes on each model span.Status = Error on failure.traceparent; on the receiving side, extract it. This is the move that keeps the house one trace.[OTEL-TRACES] [OTEL-GENAI] [W3C-TRACECTX]
BRING A SYSTEM YOU CANNOT FULLY EXPLAIN.
The best thing to instrument is the agent stack that already surprises you.
Bring: a laptop, an agent or multi-agent system you ship (or want to), and one run where you could not tell which step or which agent went wrong. You leave able to see it.
[BOARD]
ALL THREE LABS, ONE HUB.
The labs do not expire when the night ends. The hub stays up: run them again, share them with your team, hand them to the agent that ships on Monday.
[BOARD]
THREE HABITS OF AN OBSERVABLE HOUSE.
The mechanics are settled. The discipline is what separates a glass house from a pile of disconnected traces.
IF IT CROSSES A BOUNDARY, IT CARRIES CONTEXT.
One broken handoff and the trace goes dark.
Every A2A call, every queue, every async hop injects and extracts traceparent. The discipline is uniform: no agent is allowed to start a new trace just because it forgot to look for the old one. [W3C-TRACECTX]
EVERY KNOWN FAILURE HAS A SPAN SIGNAL.
Status = Error, not a wrong answer.gen_ai.usage.* so budget pressure has a name.[OTEL-TRACES] [OTEL-GENAI] [F1]
ONE RUN TO DEBUG. THOUSANDS TO STEER.
[OTEL-TRACES] [OTEL-GENAI]
THE WHOLE HOUSE, MADE OF GLASS.
By the end of the night your agents emit traces, and you can see inside every conversation between them.
Black box to glass box was one agent. The glass house is the rest: shared trace-id, propagated context, one waterfall across the whole society. You leave with the labs and the playbook. [BOARD]