feat(shadow): observe and repair Review friction safely

[joshyorko]

User promise

Shadow Mode watches how Review is used, detects friction or regressions, reproduces and repairs them in an exact-revision shadow, and returns verified improvements for explicit human approval.

Shadow Mode is one first-class Review capability with three product surfaces:

• Shadow Observer — structured evidence, deterministic detectors, correlation, grouping, schedules, budgets, and recurrence tracking.
• Shadow Repair Loop — isolated reproduction, focused patching, deterministic verification, preview, cancellation, and cleanup.
• Improvement Inbox — the maintainer's durable decision surface.

Manual Fix this capture and automatic observation are both part of this issue. Several small PRs may deliver the slices below, but #351 remains the sole owner until the complete loop is proven.

Retain Shadow Mode as the product name because it describes both observing and repairing through an isolated copy. It must never imply hidden surveillance: Review always shows the active mode, retained data, current budget, and kill switch.

Problem and thesis

Dogfooding currently produces valuable but lossy knowledge: a crash, repeated refresh, abandoned action, stale result, cleanup failure, or confusing decision path may be visible in one session without becoming a bounded reproduction or a trustworthy improvement.

A chat window beside Review cannot solve that reliably. Shadow Mode is valuable because Review already knows typed actions and states, exact source/image identity, engine lifecycle, verification commands, screenshots, and candidate history. The product primitive is:

Capture enough structured application context to reproduce a finding safely, change only an isolated exact-revision shadow, prove the candidate, and ask a human whether to promote it.

AI is optional infrastructure. Exact event capture, deterministic detection, grouping, revision correlation, reproduction-capsule creation, replay, cleanup validation, and evidence-card generation must remain useful without a model call.

flowchart LR
    R["Live Review"] --> O["Shadow Observer"]
    M["Manual Fix"] --> O
    Q["Session / Schedule / Push / Release"] --> O

    O --> P{"Worth acting on?"}

    P -->|Observe| I["Improvement Inbox"]
    P -->|Investigate| C{"Execution capacity"}

    C -->|Local| G["Local Engine"]
    C -->|Fleet| H["Hive"]

    G --> S["Sandboxed Repair"]
    H --> S

    S --> I

    I --> A{"Maintainer Review"}

    A -->|Approve| W["Issue / Draft PR"]
    A -->|Not now| D["Snooze / Mute / Reject"]

    R --> A

Loading

The diagram describes required capabilities. #362 is the canonical public Review execution-engine contract; #351 consumes #362 and does not define a competing engine.

Authority and adoption boundary

Review always owns:

• the maintainer-facing client, Shadow Mode policy, local evidence inventory, Improvement Inbox, previews, and human confirmation;
• finding identity and state;
• exact source/image provenance and candidate trust receipts;
• the decision to discard, snooze, mute, investigate, open an issue, or prepare a draft PR.

The Dagger-style engine owns:

• sandbox allocation, execution, event streaming, cancellation, artifact transfer, and cleanup;
• identical execution semantics for local and later remote capacity.

Shadow Mode consumes that engine; it is not a second execution backend.

Hive remains authoritative for:

• fleet-scale task selection, assignment, contributor coordination, remote capacity, and unattended work;
• deciding which Hive work is admitted and assigned.

Standalone Review must work for one local repository without deploying Hive. When multiple repositories, concurrent repairs, remote compute, or unattended capacity become the user's problem, the same Review policy, evidence schema, Inbox, receipt, and human gate may use Hive-coordinated capacity. Review does not become a mini-Hive, rank Hive work, or invent a second queue.

No current Hive API for Shadow repairs is claimed. Hive-backed execution is a future graduation contract that requires a public capability and the same trust receipt; it is not implementation authorization in this issue.

MCP may later expose the same capability externally, but remains an adapter to the same Review policy and engine boundary—not a control plane or execution backend.

Operating modes and controls

• off — default on first run. No background observation or model calls; manual Fix this remains available.
• observe — collect permitted structured evidence, run deterministic detectors, deduplicate findings, and create improvement cards. Model analysis runs only within an explicit budget.
• repair — explicit opt-in. High-confidence findings may automatically start one bounded Shadow Repair Loop and prepare a verified candidate.

No mode may autonomously publish an issue, push a branch, create a PR, approve, merge, deploy, release, promote a candidate, or modify the live Review session.

The user can immediately switch to off, cancel a repair, snooze a card, mute a detector/fingerprint, discard evidence/candidates, and inspect current retention and spend.

Initial safe-default hypotheses to validate in the first slice:

• one concurrent repair;
• one automatic attempt per finding fingerprint and exact revision before human action;
• deterministic detectors before any model;
• background model analysis disabled until a token/runtime budget is set;
• scheduled aggregation disabled until configured;
• local retention bounded by both age and bytes;
• automatic investigation limited to proven high-confidence classes until dogfood evidence expands the allowlist.

Exact numeric retention, idle, cooldown, confidence, and budget defaults must be explicit, configurable, and tested in the owning slice; this issue does not hard-code a cron schedule.

Trigger model

Shadow Observer supports:

1. Manual — Fix this, with optional note and screenshot.
2. Immediate deterministic — crash, timeout, leaked resource, invalid transition, cleanup failure, repeated command failure, or security-invariant failure.
3. Session/idle — compact analysis at session completion or after a bounded idle period.
4. Scheduled aggregation — repeated friction across sessions.
5. Revision/release — compare a new commit, push, image, or release with the preceding exact identity.
6. Candidate replay — replay the originating reproduction against a candidate.
7. Recurrence — observe post-push/post-release sessions for the same fingerprint.

Sampling, deduplication, cooldowns, confidence thresholds, repair concurrency, runtime limits, token budgets, and cheap-model routing are policy inputs. Cache semantic analysis by exact evidence identity. Equivalent findings roll into one card with occurrence count, affected revisions, and newest evidence; they never launch duplicate repair agents.

Evidence contract

Use a small, schema-versioned Review event envelope that can map to OpenTelemetry traces, spans, events, logs, and resource identity where that improves interoperability. Do not require a collector or hosted telemetry service. Do not use OTel baggage for sensitive values.

Minimum fields:

• schema version and event ID;
• exact Review commit and image digest when available;
• repository/worktree identity without source contents;
• session, engine-job, ReviewRun, trace, and parent-operation IDs;
• backend and harness identity;
• typed screen, action, state, and terminal transitions;
• monotonic timing, duration, retry, cancellation, timeout, and cleanup outcome;
• verification/test receipt handles;
• explicit maintainer note and optional screenshot reference;
• redaction result, retention class, and trust class.

Evidence is local-first, bounded, redacted before persistence or model access, retention-limited, and integrity-bound to its exact revision. Do not indiscriminately record raw keystrokes, prompts, credentials, source code, diffs, terminal contents, clipboard data, or production data.

A reproduction capsule contains only the minimum redacted events, deterministic fixture/state, exact identities, and permitted artifact handles needed to replay the finding. Logs, screenshots, repository text, and external issue content are untrusted evidence—not instructions or authority.

Finding state model

Every card distinguishes:

• Observed — typed evidence proves an event occurred.
• Inferred — bounded analysis suspects friction; this is not proof.
• Reproduced — the exact-revision shadow reproduced the finding.
• Patched — a focused candidate exists.
• Verified — required deterministic checks and replay passed in isolation.
• Promoted — a human approved an exact-head GitHub action.
• Resolved after push — replay or later-session evidence shows no recurrence within the declared observation window.

Also represent pushed-not-observed, recurring, invalidated-by-revision, and inconclusive. A merge never silently means resolved, and no stage may be skipped or inferred from a later stage.

Shadow Observer

Shadow Observer:

• validates and redacts structured events;
• runs deterministic detectors without a model;
• groups by a documented fingerprint and records recurrence;
• correlates sessions, engine jobs, ReviewRuns, commits, images, candidates, and releases;
• creates reproduction capsules and evidence-only cards;
• invokes compact semantic analysis only for ambiguous friction within policy;
• explains which rule/model detected the finding, what data it used, confidence, cost, and why policy chose observe or investigate;
• enforces sampling, retention, cooldown, deduplication, concurrency, and spend limits.

A model-generated suspicion remains Inferred until deterministic evidence or shadow reproduction promotes it.

Shadow Repair Loop

When policy authorizes investigation, the engine must:

1. Allocate a disposable sandbox at the exact affected Review revision and image identity.
2. Prove the required gVisor/runsc boundary before credentials or agent-controlled work enter it.
3. Provide only the bounded, redacted reproduction capsule.
4. Create a separate checkout/worktree and writable candidate state.
5. Mount no writable live Review state and pass no live Hive assignment token or production credential.
6. Reproduce before editing; insufficient reproduction returns an evidence-only card.
7. Produce one focused patch.
8. Run verification appropriate to the affected surface.
9. For TUI changes, drive the real app with Textual Pilot and attach before/after visual evidence.
10. Return the candidate artifact and complete trust receipt.
11. Clean up on success, failure, cancellation, timeout, interruption, or client loss without disturbing live Review.

#348 and merged PR #349 (merge commit a434077525e36e53f9dcc4fbdd435ab74ab39dd6, head 1a117ffa1aaa782abf528d9ad92e8e7c222af928)'s fail-closed contract remains mandatory wherever runtime ownership moves: positive runsc proof, no fallback to crun/runc, proof before credential transfer, exact-identity cleanup, and coverage of every agent-capable execution path. The merged #349 fail-closed isolation contract and tests must survive any engine relocation.

The running Review instance is never patched or hot-reloaded. An approved candidate is deliberately restarted or published through the normal exact-head workflow.

Improvement Inbox and trust receipt

A concise default card, with expandable evidence, shows:

• problem statement, severity, confidence, and current finding state;
• detector/analysis and why it fired;
• occurrence count and affected exact revisions/images;
• observed versus inferred facts;
• bounded evidence and redaction inventory;
• investigation and reproduction result;
• candidate revision, changed files, and diff summary;
• tests, replay, and verification performed/not performed;
• before/after screenshot or equivalent preview for visual defects;
• backend/model provenance when used;
• token, runtime, cache, and remote-capacity cost;
• cleanup result;
• recurrence status after later pushes;
• actions: preview, investigate, cancel, open issue, prepare draft PR, snooze, mute fingerprint, discard.

A card may exist without a candidate. Incomplete, stale, unverified, cleanup-failed, or redaction-failed work must be visibly non-promotable.

Human promotion and stale-head protection

Every GitHub write uses #184's immutable ActionPlan and explicit human-confirmation boundary. The plan binds actor, repository, current head, action, final body, and exact operations, then revalidates mutable state immediately before execution.

No model-only path may push, create an issue or PR, approve, merge, deploy, release, or promote. Candidate substitution, changed heads, missing verification, missing runtime proof, or cleanup failure invalidates promotion.

After a push or new image, Shadow Observer may replay the original capsule and watch the declared post-push window. It reports resolved, recurring, invalidated, or inconclusive with exact evidence.

Security and privacy threat model

Threat / asset	Required mitigation
Live credentials, Hive authority, writable state	Never enter the shadow; allowlisted inputs only; redact before persistence/model access
Malicious repository text, logs, screenshots, or prompts	Treat as untrusted evidence; schema validation; no tool/mutation authority; bounded rendering
Poisoned or tampered events/capsules	Schema version, exact provenance, integrity identity, trust class, rejected unknown fields
Sandbox escape or weaker runtime fallback	Fail-closed gVisor/runsc proof from #348/#349; no repair or credential transfer on failure
Candidate substitution or stale-head promotion	Exact candidate/source/image identity; immutable ActionPlan; immediate revalidation
Cleanup failure and orphaned resources	Engine-owned bounded cancellation/cleanup; exact identity; orphan detection; non-promotable receipt
Privacy leakage and surveillance	off default; local-first bounded inventory; no raw recording; retention limits; preview/delete/mute
Denial-of-wallet or repair storms	Hard token/runtime budgets; sampling; grouping; cooldown; concurrency 1 default; kill switch
Self-repair loops and benchmark overfitting	Reproduce before edit; focused patch; independent deterministic checks; post-push recurrence; attempt cap

Cost and success evidence

Every session/card reports whether a model ran and its token, runtime, cache, and remote-capacity cost. Measure:

• time from first occurrence to useful finding;
• deterministic share of findings;
• reproduction and candidate-verification rates;
• candidate acceptance and post-push recurrence rates;
• false-positive and duplicate rates;
• maintainer attention consumed versus time saved;
• cost per observed session, investigated finding, and accepted repair;
• cleanup success and credential/redaction failures;
• snooze, mute, discard, and Shadow Mode disable rates;
• parity of local and Hive-backed receipts.

Do not enable automatic repair for a detector class until its documented dogfood evidence shows acceptable false-positive, cleanup, privacy, attention, and cost behavior.

Kill or redesign the automatic path if any of these persist beyond a bounded repair:

• credential/redaction failure;
• unverified runtime or cleanup residue;
• candidate substitution or stale promotion;
• duplicate/false-positive volume that costs more attention than it saves;
• repeated repairs of previous Shadow repairs;
• budget enforcement failure;
• candidates that pass local checks but repeatedly recur after promotion.

off, manual capture, deterministic evidence, and the Inbox must remain usable even if semantic analysis or automatic repair is disabled.

Delivery slices and checkpoints

No engine-dependent implementation starts until the minimum #362 engine slice required by Shadow Mode is merged. After that gate, land independently reviewable PRs under this one issue:

1. Evidence foundation — versioned envelope, local bounded store, redaction, integrity/provenance, retention, off|observe|repair, kill switch.
   Checkpoint: fixture/schema migration, secret corpus, retention/erase, malformed/untrusted event, and zero-model tests.
2. Observer and Inbox — manual capture, deterministic detectors, grouping, cooldown, evidence-only cards, snooze/mute/discard.
   Checkpoint: real TUI/Pilot journey; duplicate and false-positive fixtures; no repair or GitHub write.
3. Bounded semantic analysis — session/idle/scheduled analysis, sampling, cache, confidence, token/runtime budgets.
   Checkpoint: hard budget exhaustion, model-unavailable fallback, explainability, and cost receipts.
4. Shadow Repair Loop — engine adapter, exact-revision capsule, gVisor proof, reproduction-first patching, deterministic verification, artifact/receipt return, cancellation/cleanup.
   Checkpoint: missing/unverified runsc, prompt-injection, credential absence, timeout/interruption/orphan, and visual before/after tests.
5. Automatic investigation — high-confidence allowlist under repair, concurrency and attempt caps.
   Checkpoint: policy matrix, repair-storm prevention, manual cancellation, no autonomous GitHub mutation.
6. Revision correlation and scale — push/image/release comparison, replay, recurrence states, identical local/Hive-backed receipt.
   Checkpoint: stale/changed head, pushed-not-observed, resolved, recurring, invalidated, inconclusive, and Hive-authority tests.

Likely hot boundaries are the shared TUI action/state registry, ReviewRun/evidence/ActionPlan contracts, engine adapter/event boundary, local state/retention, Improvement Inbox presentation, Pilot fixtures, and #349's launcher/security boundary. Each slice must update the closest canonical skill/model; do not create a competing architecture document.

Work may proceed in parallel only where write sets and contracts are disjoint after the engine boundary is public. #351 stays open through the complete dogfood acceptance.

Acceptance criteria

• Shadow Mode has one coherent user promise and the named Shadow Observer, Shadow Repair Loop, and Improvement Inbox surfaces.
• off, observe, and repair behave as documented; manual Fix this works in every mode.
• Standalone Review works for one local repository without Hive.
• Optional Hive-backed capacity preserves the same evidence, receipt, Inbox, and human authority while Hive retains fleet selection/assignment.
• Structured evidence is local-first, bounded, redacted before persistence/model access, schema-versioned, retention-limited, integrity-bound, and tied to exact source/image identity.
• Raw keystrokes, prompts, credentials, source, terminal contents, clipboard, and production data are not indiscriminately recorded.
• Deterministic failures, grouping, release correlation, capsule creation, replay, cleanup validation, and evidence-only cards work with zero model calls.
• Session/idle/scheduled analysis obeys sampling, cooldown, confidence, concurrency, token, runtime, cache, and retention policy.
• Equivalent findings deduplicate into one card with occurrence/revision history.
• Every card distinguishes observed, inferred, reproduced, patched, verified, promoted, and resolved-after-push states.
• High-confidence allowlisted findings may start one bounded Shadow Repair Loop only under repair.
• Missing or unverified runsc fails closed before agent work or credential transfer.
• The shadow receives no writable live state, live Hive assignment token, or production credential.
• Repair reproduces before editing and returns a focused exact-revision candidate with a complete trust receipt.
• TUI changes pass Textual Pilot and include before/after visual evidence.
• Failure, cancellation, timeout, interruption, or client loss removes all disposable resources and cannot disturb live Review.
• No GitHub write or promotion occurs without exact-head human confirmation and current-state revalidation.
• A later push/image replay reports pushed-not-observed, resolved, recurring, invalidated, or inconclusive; merge alone never means resolved.
• Cost, cleanup, redaction, detector, confidence, and recurrence evidence is visible and testable.
• Kill switch, off, snooze, mute, cancel, discard, and evidence deletion work.
• The full loop is dogfooded on real Review failure classes from detection through a human-approved draft PR, including at least one deterministic no-model finding and one visual TUI finding.
• Kill/redesign criteria are evaluated before automatic repair is admitted beyond the initial allowlist.

Non-goals

• Hidden cloud telemetry, raw keylogging, or indiscriminate session replay.
• A hosted telemetry dependency, Kubernetes requirement, or permanently privileged sidecar.
• Generic IDE/devcontainer management or production debugging with copied state.
• Hot-patching or silently modifying live Review.
• Autonomous issue/branch/PR creation, approval, merge, deployment, release, or promotion.
• A second scheduler, task queue, Hive replacement, or execution control plane.
• Changing backend selection (feat(worker): consume Hive backend selection instead of enforcing Goose + Copilot locally #167) or Goose task transport (design: drive Goose over ACP instead of scraping the contributor pane #119).
• Treating model suspicion, a generated patch, green unit tests, or a merged PR as proof of resolution.
• Unlimited inference, retention, attempts, or concurrent repair agents.

Relationships and reusable contracts

• Review V2 product parent: feat(v2): ship Review V2 as a harness-neutral MCP App review cockpit #173
• Backend-neutral result cockpit: PR feat: add backend-neutral review result cockpit #180
• Exact-head evidence manifest: feat(core): bind every review harness to one exact-head evidence manifest #183 / PR feat(core): add bounded review evidence manifest #190
• Human-confirmed exact-head mutations: feat(core): make every GitHub mutation use one exact-head ActionPlan #184 / PR feat(core): make every GitHub mutation use one exact-head ActionPlan #191
• Optional future MCP adapter: feat(mcp): ship the minimal Factory → Queue → Review App over the shared core #188
• TUI/Pilot product surface: feat(dashboard): make the Review TUI a cohesive, responsive cockpit #196 / PR feat(dashboard): add semantic view foundation #197
• Exact-head re-review/replay precedent: feat(review): focus re-review on the head delta without carrying approval forward #185 / open PR feat(tui): present exact-head re-review deltas #350
• ReviewRun lifecycle: feat(harness): make review runs re-entrant instead of process-lifetime-bound #255 / merged PR feat(harness): add ReviewRun re-entry state machine with resumable capability #347
• Runtime restoration baseline: Restore Goose/Hive runtime on the direct lab-runner fork #346
• gVisor security contract: feat(security): require gVisor isolation for Review agent containers #348 and merged PR feat(security): require gVisor isolation for Review agent containers #349 (merge commit a434077525e36e53f9dcc4fbdd435ab74ab39dd6, head 1a117ffa1aaa782abf528d9ad92e8e7c222af928)
• Shared provisioning relationship: projectbluefin/common#1038 / PR #1039; native Bluefin acceptance remains feat(security): require gVisor isolation for Review agent containers #348 -> projectbluefin/bluefin#1139 (open).
• Hive worker-backend boundary: feat(worker): consume Hive backend selection instead of enforcing Goose + Copilot locally #167
• Goose ACP transport boundary: design: drive Goose over ACP instead of scraping the contributor pane #119

Public engine dependency: blocked until the minimum projectbluefin/review#362 engine slice required by Shadow Mode is merged. #362 owns the Review execution engine; #351 consumes #362 and does not define a competing engine. Shared runsc provisioning is owned by projectbluefin/common#1038 / PR #1039, while native Bluefin acceptance remains projectbluefin/bluefin#1139.

Primary design evidence

• Dagger Engine configuration, ephemeral services, and OpenTelemetry-powered observability
• Sentry issue grouping, Replay privacy, and Autofix stopping points
• OpenTelemetry traces, sampling, and baggage security
• Textual Pilot and snapshot testing
• gVisor rootless execution and Podman runtime inspection

Classification

SAFE-FOR-AGENT in independently reviewable slices after the public engine contract lands and is reconciled