C28
Agent Observability
7 lessons3.5 hoursPlannedFree
Outcome
Trace, debug, and monitor LLM agents and multi-agent systems in production.
Prerequisites
- Required: C16 LLM Agents and Tool Calling
- Recommended: C27 LLM Evaluation and LLM-as-a-Judge
Lessons
- L01Why agents need tracing: non-determinism and multi-step failures · coming soonExercise: List five questions about an agent run you can't answer from logs alone.
- L02Traces and spans for LLM and tool calls (OpenTelemetry GenAI conventions) · coming soonExercise: Instrument an agent with OpenTelemetry.
- L03Tracing platforms: Langfuse, Arize Phoenix, and self-hosted options · coming soonExercise: Send traces to a tracing backend and inspect one run end to end.
- L04Token, cost, and latency accounting per task · coming soonExercise: Build a per-task cost and latency breakdown across agents.
- L05Debugging multi-agent runs from traces · coming soonExercise: Find the root cause of a looping agent from its trace.
- L06Online evaluation and feedback loops · coming soonExercise: Sample production traces, score them with an LLM judge, and add failures to the eval set.
- L07Dashboards, SLOs, and alerts for agent systems · coming soonExercise: Define SLOs and alert rules for an agent service.