A production runtime must define what happens after timeouts, invalid output, denied tools, partial side effects, duplicate execution, budget exhaustion, and evidence failures.
Failure and recovery state machine
Text description
A state machine begins at admitted, then executing, validating, and completed. Failures branch to classified. A transient and idempotent failure can enter delayed retry and return to executing. An authorization or policy denial ends as denied. A partial side effect enters reconcile, then either compensated, completed with warning, or escalated. Budget exhaustion and human-review timeout end as suspended. Every terminal state writes evidence, and duplicate requests return the prior result when the idempotency record is complete.
Recovery rules
- Detect and classify the failure.
- Determine whether retry is safe and whether an idempotency key is required.
- Preserve or restore known-good state.
- Compensate completed side effects where rollback is impossible.
- Escalate to a human or incident path when authority or evidence is insufficient.
- Persist minimized evidence of decisions, attempts, effects, and recovery.
Use the reference architecture failure matrix for failure-specific behavior.