← Back to the atlas
AWS & ML

Debugging systems with several plausible explanations

In progress Draft date

A bad result might come from the model, the context, the orchestration, or the infrastructure — debugging starts by separating those possibilities

Working outline — full essay forthcoming

Rough outline

Build a hypothesis tree

  • List competing explanations before changing prompts or model settings
  • Identify the evidence that would distinguish each explanation

Instrument the right boundaries

  • Capture retrieved context, tool arguments, model responses, and timing
  • Make missing evidence visible instead of assuming a successful handoff

Reduce the problem

  • Replay with fixed tool responses or a smaller context
  • Change one boundary at a time and retain the failing case

Turn the diagnosis into protection

  • Add a regression case at the layer responsible for the failure
  • Document what the test proves and what remains uncertain