Cookbook: Agent Trajectory Evaluation
AgentTracer ContextVar spans + post-hoc criteria.agent (one evaluate, zero extra LLM localization).
Cookbook 3 — Agent Trajectory Evaluation
Realtime spans (AgentTracer)
AgentTracer / @agent_tracer collect spans in-memory with contextvars isolation.
auto_evaluate=False is mandatory — call evaluate_trace explicitly (one /evaluate, no surprise credits).
from qurator import Qurator, AgentTracer, agent_tracer
client = Qurator()
with AgentTracer(name="support_agent") as tracer:
tracer.add_message("user", "Where is order 99?")
@agent_tracer("search_orders", kind="tool")
def search_orders(order_id: int) -> dict:
tracer.record_tool("search_orders", arguments={"id": order_id}, result={"status": "ok"})
return {"status": "ok"}
search_orders(99)
tracer.add_message("assistant", "Order 99 has shipped.", name="support_agent")
breakdown = tracer.evaluate_trace(
client,
criteria=["Passes if the agent recovers after a tool timeout."],
anonymize=True,
)
# Zero extra LLM calls — buckets from turn-localized diagnostics:
breakdown.results_by_agent_name
breakdown.results_by_tool_nameNo OpenAI SDK monkey-patching. Prefer AgentTracer.from_openai_messages([...]) for post-hoc exports.
Post-hoc pack / Studio
Studio pack agent_trajectory_assurance_v1 loads a sample tool-retry trajectory into Visual Builder.