Testing & agents
A useful failure needs a replay
Rough outline
Define a useful failure record
- Capture task inputs, context, tool responses, versions, and event ordering
- Separate sensitive data from the diagnostic information that must be retained
Distinguish replay from reproduction
- Explain what can be replayed exactly and what may vary across model calls
- Record the limits of the evidence rather than promising determinism
Design the investigation workflow
- Freeze external responses and inspect intermediate decisions
- Minimize the case until the causal boundary becomes clear
Make replay part of development
- Attach a compact replay case to a bug report
- Turn a resolved failure into an appropriately scoped regression check