> ## Documentation Index
> Fetch the complete documentation index at: https://docs.blobhub.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Recovery

> Behavior across worker restarts, agent crashes, and network blips

A `session_agent_harness` section is designed so that **a clean restart picks up exactly where it
left off**. The authoritative state is split between server-side envelopes (handoff) and local
YAML files (local recovery), and the worker writes the local file **before** any externally
observable side-effect.

## Worker restart

A graceful shutdown (`SIGINT`, `SIGTERM`, `q` in the TUI) leaves:

* `instance.yaml` removed.
* Each active thread's local `thread.yaml` snapshotted at the last completed turn boundary.

The `worker` object is left in place (its `instance.status` only ever holds `attached`; `last_seen_at`
goes stale) and is refreshed on the next attach.

An ungraceful exit (`SIGKILL`, OS crash) skips those steps. The next `start` handles both cases.

On the next `start`, for each section:

1. The instance lock check accepts a dead-PID `instance.yaml` and replaces it.
2. Credentials and identity are re-verified against `/v1/users/me`.
3. The `worker` session object is read; if `metadata.user.user_id` matches, the section re-attaches
   (overwriting `instance` with our fresh block). If it differs, the section refuses with
   `SESSION_OWNED_BY_DIFFERENT_USER` (see [Worker object](/worker/session-agent-harness/worker-object)).
4. The session-event poll resumes from `events.last_processed_at`.

## Per-thread recovery

A thread is recoverable only when **both** views agree it was running: the persisted local `thread.yaml`
shows `agent.state == "active"` and the server envelope shows `instance.state == "active"`. (The local
`thread.yaml` keeps the agent state under `agent.state`; the wire envelope keeps it under
`instance.state` — see [Job Session Object](/worker/session-agent-harness/thread-object).) For each such
thread, the worker runs:

1. Re-validate `workspace.work_folder`, `agent.permissions`, and — for a `guarded` thread — its `policy`
   block, re-read fresh from the envelope (the same checks as initial activation; see
   [Job Session Object → Policy](/worker/session-agent-harness/thread-object)). A policy failure here
   fails recovery with the same code activation would have raised (`POLICY_MISSING`, `POLICY_INVALID`, or
   `POLICY_AUTONOMOUS_FORBIDDEN`).
2. Re-derive the `artifacts` block from the envelope and re-create the outbox if it carries one.
   `artifacts` (like `policy`) is never mirrored into `thread.yaml` — it is in-memory only for the life of
   the turn loop, so a freshly reconstructed agent after a restart must re-read it or a thread that was
   publishing before the restart would silently stop after one turn.
3. Acquire a concurrency slot.
4. **Resume the agent session** via the `codeagents` SDK using the persisted `agent_session_id`. If the
   resume fails, the thread transitions to `failed` with `AGENT_CRASHED` (a failed resume is reported the
   same way as any other agent crash; there is no separate resume error code).
5. Cancel any leftover `pending_prompt` by posting `pending_prompt_resolved` with `reason:
   worker_restart` (see [Interactive prompts](/worker/session-agent-harness/interactive-prompts#worker-restart)).
6. Replay thread items posted while the worker was down: fetch
   `list_session_thread_items(created_since = items.last_consumed.created_at)`, drop self items,
   feed remaining user items as the first post-recovery turn's prompt.
7. Post a `thread_recovered` activity-log item on the `worker` thread.
8. Resume normal active behavior — including posting a `policy_applied` item before the next turn, for a
   `guarded` thread.

Recovery runs **before** activation of any new `pending` threads on the same section, so threads
already running keep their concurrency slots; new pending work waits if the configured
`concurrency.max_agents` ceiling is full.

### If max\_agents was lowered across restart

If you reduced `concurrency.max_agents` and there are more recoverable active threads than slots,
the overflow are logged as `thread_recover_deferred` and remain inert until a future restart with
enough slots. They aren't transitioned to `failed`.

## Agent crash (worker still running)

An agent process that exits unexpectedly transitions its thread to `failed` with `AGENT_CRASHED`. The
worker writes `instance.state = "failed"` on the envelope and records the error code and message in the
local `thread.yaml`; the error is **not** placed on the envelope. The failure surfaces server-side through
a `thread_failed` activity item (carrying the code and message) on the `worker` thread. The worker does
**not** auto-retry. To resume, the user updates the envelope to `instance.state = "pending"` (see
[Handoff](/worker/session-agent-harness/thread-handoff#failed--pending-reset)).

## Network blips

Transient errors from `api.blobhub.io` (429, 5xx, connection timeouts) are classified as
`API_RATE_LIMITED` / `API_TRANSIENT_ERROR` / `API_NETWORK_ERROR`, logged, and retried with
exponential backoff up to `polling.backoff_max_ms` (default 30 s). They do **not** change any state
machine. In the TUI they surface as a warning ribbon; in headless mode they appear in the JSON log.

## What does not survive a restart

* The in-memory inbound queue beyond `items.last_consumed` (replayed from the server).
* In-flight agent turn state beyond the resumable `agent_session_id` (recovered by the SDK).
* Agent stdout that hadn't yet been posted — only present in the local `thread.log` (not on BlobHub).
* An unresolved interactive prompt — cancelled on recovery.

## Detachment while running

If someone deletes the `worker` session object (e.g. via `delete_session_object` from the API or
playground), the worker observes the `session_object_deleted` event and stops that section with
`SESSION_DETACHED_EXTERNALLY`. Other sections continue running. All `ThreadAgent`s for that section
shut down cleanly (the codeagents SDK is asked to cancel each session); their threads remain in
whatever state they were in on the server and the worker reattaches if you re-create the `worker`
marker.

## Session closed or deleted

A section also stops when the session itself reaches the end of its
[lifecycle](/blob-types/workflow/workflows/session-lifecycle). The worker watches for the session
lifecycle events on the same event stream it already polls:

| Event              | Worker behavior                                                                 |
| :----------------- | :------------------------------------------------------------------------------ |
| `session_closing`  | Stops starting new thread work. In-flight turns are left alone; stays attached. |
| `session_closed`   | Detaches the section with `SESSION_CLOSED`.                                     |
| `session_deleting` | Detaches the section with `SESSION_DELETED`.                                    |
| `session_deleted`  | Detaches the section with `SESSION_DELETED`.                                    |

`session_deleting` is treated as terminal rather than waiting for `session_deleted`: a session
deleted directly from `open` never emits `session_closed`, and its data is about to disappear.

On detach every `ThreadAgent` in the section is torn down and its concurrency slot released. This is
a cancellation, not a drain — a closed or deleting session rejects every write, so an in-flight turn
could not post its result even if it were allowed to finish. Other sections keep running.

<Note>
  The worker does not release its `worker` object on this path. Once the session leaves `open` it
  rejects writes, so the marker cannot be updated; a deleted session takes it with everything else. A
  closed session that is later reopened still carries the marker, and the worker reattaches on its
  next start.
</Note>

If the session becomes unreadable altogether — the poll starts returning `403`, which is what a
deleted session answers — the section detaches with `SESSION_UNREACHABLE` rather than failing
silently.

## See also

* [Session Lifecycle](/blob-types/workflow/workflows/session-lifecycle)
* [The `worker` session object](/worker/session-agent-harness/worker-object)
* [Job Session Object](/worker/session-agent-harness/thread-object) — the `artifacts` and `policy` blocks
  re-derived on every recovery
* [Handoff](/worker/session-agent-harness/thread-handoff)
* [Interactive](/worker/session-agent-harness/interactive-prompts)
* [Reference](/worker/session-agent-harness/reference)