Failure detection and recovery

How agents detect their own failures

Driftless agents do not wait for a human to tell them something went wrong. Failures surface from the tool output the agent is already reading, in the same step that produced them.

• Failing tests. Agents run the tests for the code they change and read the results directly. A failing test is the agent's signal to diagnose and fix, in the same working session.
• Type errors and lint failures. Type checkers and linters run against changed code. An error in the output is treated as a defect to resolve before the work can ship.
• Failed CI checks. Build, lint, and test pipelines run on every change. When a check fails, the failure text lands in the agent's context and the agent acts on it without being asked.
• Deployment verification. Work that changes running infrastructure is verified against the live endpoint after the change lands, so a failed rollout is caught by the agent's own check rather than by a user.

No escalation step exists between failure and action. The agent that ran the failing command is the same agent that reads the failure and starts recovering from it.

Retry with a changed strategy

When a fix attempt fails, the next attempt uses a different approach. Repeating the same change and expecting a different result is the anti-pattern; the workflow treats it as one.

• The failure changes the plan. A failed attempt is evidence about the problem. The agent reads what the failure actually said before choosing the next move, so the second attempt addresses the diagnosed cause rather than the first guess.
• Different tools for the second attempt. If an edit did not resolve the failure, the next attempt reaches for different information: the error output itself, the documentation of the system that reported the error, or a direct probe of the failing endpoint. The strategy changes because the evidence points somewhere new.
• Verified, not assumed. A recovery only counts when the original check that failed now passes. The agent re-runs the same verification that caught the failure and ships nothing on the strength of an assumption.

The published trace at the end of this page shows the full loop: a first strategy that failed verification, a diagnosis pulled from the failure output, and a second strategy driven by that diagnosis.

The blocked-state protocol

Some failures are not the agent's to fix. When reasonable fix attempts are exhausted, or when the unblocking step requires access the agent does not have, the agent stops rather than loops.

• The blocker is recorded. What was attempted, what the failures said, and why the remaining step is out of reach are written onto the task, so the next reader starts from the diagnosis rather than from zero.
• It is routed to a human. The routing includes concrete instructions: the exact step that requires operator access and how to verify the result afterwards. The task moves to a blocked state where a human picks it up.
• Unrecoverable does not mean lost. Work the agent cannot finish is handed off with its full context intact. The published trace below includes exactly this moment: the agent identified a vendor dashboard step that only an operator with account access could perform and routed it with instructions instead of retrying against a wall it could not move.

Cross-session recovery

Agents run in sessions, and sessions end: a process can crash, a host can restart, a network path can drop. Driftless detects this and recovers the work without anyone filing a ticket.

• Heartbeats while working. Every task an agent is actively working on emits a regular heartbeat. A healthy heartbeat says the work session is alive.
• Stale detection. If a heartbeat stops arriving, the task is marked stale. The absence of the heartbeat, checked continuously, is the signal.
• Automatic re-dispatch. A stale task is re-dispatched to an available agent automatically, so an interrupted session does not strand work in progress. The recovered task keeps its acceptance criteria and history, and the picking-up agent starts from the recorded state.

A real recovery, end to end

The following trace is one thread from recorded task history, on the work that put our public status page on its own subdomain. It contains a changed-strategy retry and a blocked-state routing in the same arc, and no human typed anything before the recovery actions described here. The full decision log with timestamps is retained on the task board; the narrative below is scrubbed to capability level.

1. The goal. The public status page, hosted by our monitoring vendor, needed to serve from a subdomain of our own domain so that evaluators and customers could find it as first-party infrastructure. The agent added the required DNS alias for the vendor's service and deployed it.
2. The failure signal, unprompted. The deployment verification checked the live endpoint and the check failed: the address did not serve over HTTPS. The agent queried the endpoint directly and read the error from the response. The vendor's edge network returned its "hostname not configured" error (error 1001), which said the DNS record existed but the vendor platform did not yet recognize the custom domain. The failure surfaced from tool output, with nobody watching.
3. Changed strategy, not a repeated one. The first strategy, fixing the DNS configuration and redeploying, had already been executed and had not resolved the failure. Instead of re-running it, the agent switched approach: it consulted the vendor's custom-subdomain documentation and matched the observed error against it. The documentation showed a second required step the agent's first pass had not covered.
4. Diagnosis and routing. That second step lives in the vendor's dashboard, which requires account access the agent does not hold. The agent recorded the blocker on the task with the exact dashboard path, the field to set, and a verification command for after the fix, and stopped working on the thread rather than retrying against a permission boundary.
5. The correction arrives. The operator completed the dashboard step and, in the same pass, corrected one detail in the DNS target the agent had configured, following the vendor's updated instructions.
6. Verified recovery. The agent applied the corrected target, deployed, and re-verified end to end: the subdomain resolved to the vendor's service, served the status page over HTTPS with a valid certificate, and returned the expected success response. The same check that had failed at the start now passed. The task closed with no further changes needed.

That is the full loop: failure detected by the agent itself, a second strategy chosen from what the failure actually said, an honest handoff at the access boundary, and a verified recovery. It is also, deliberately, an example where part of the fix belonged to a human: the system's value is that the agent got the work to exactly that point, with the diagnosis and instructions waiting, instead of looping or hiding the failure.

What this page claims and what it does not

Every statement above describes shipped behavior: self-detection of failures from tool output, retry with a changed strategy, blocked-state routing, and heartbeat-based stale-task recovery. The page does not claim unsupervised autonomous goal-setting; direction stays with people, and agents recover the execution.