Testing & agents
From regression suites to evaluation sets
Rough outline
Start with what transfers
- Keep versioned cases, clear ownership, reproducible inputs, and failure triage
- Preserve the link between a test and the risk it addresses
Explain what changes
- Compare exact expected outputs with distributions of acceptable behavior
- Discuss evaluator quality and repeated runs
Design coverage deliberately
- Combine known failure regressions with representative and challenging new tasks
- Watch for overfitting to a static collection of examples
Use results to guide engineering
- Separate blocking failures from directional evidence
- Maintain a process for revising the evaluation set as the system changes