feat(core): emit a runbook and per-step instructions for orchestrated migrate runs - #36766
Draft
leosvelperez wants to merge 23 commits into
Draft
feat(core): emit a runbook and per-step instructions for orchestrated migrate runs#36766leosvelperez wants to merge 23 commits into
leosvelperez wants to merge 23 commits into
Conversation
✅ Deploy Preview for nx-dev ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Deploy Preview for nx-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Contributor
|
View your CI Pipeline Execution ↗ for commit 252f126
☁️ Nx Cloud last updated this comment at |
![nx-cloud[bot]](https://avatars.githubusercontent.com/in/80458?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
leosvelperez
force-pushed
the
nxc-4627
branch
2 times, most recently
from
August 25, 2026 09:49
160b0e1 to
4770e39
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
…n state Additive, read-validated v1 fields for the orchestrated migrate flow: a persisted validation policy and runbook name on the run, an issues ledger with commit attribution, the kind of agent work a step awaits, and a no-progress streak record. No writers yet; the flows that use them land next.
The runbook is the self-sufficient contract a master agent session drives an orchestrated migrate run by: the re-anchor invariant on the reconcile command, the loop discipline, agent-work scope rules, the handoff contract with master-specific termination rules, commit ownership, and per-agent subagent guidance with a run-scoped handoff grant pattern. It renders from shared semantic fragments also used by the runner prompt builders, so the two surfaces cannot drift. A gateway emitter frames it as an <nx_migrate_runbook> block, neutralizing tampered lines that could forge block boundaries across the full line-terminator set. Content fixes surfaced while sharing the fragments, applying to the runner prompts too: formatting is instructed via Prettier scoped to the exact changed files instead of nx format:write (which selects the branch delta and always reformats root configs), target discovery makes nx show project --json authoritative over reading project files (which miss plugin-inferred targets), and affected verification is scoped with --files instead of the bare form. Not wired into the orchestrator yet; init and resume adopt the runbook in the next commit.
…nitializes Init and resume now answer with the run's runbook and an initialized block whose next command is the reconcile, dispensing no migration step: the contract always lands before the first command does. The runbook is written inside the run-creation lock (before run.json makes the run discoverable) and re-emitted from its stored bytes on resume, keeping the contract version-locked across the nx upgrades a run typically performs mid-flight. The run state records the resolved validation policy and the runbook name, and every step dispense and rejected step action carries a footer pointing back at the runbook and the reconcile command. The reconcile path guards on the contract: it reads, repairs, or refuses the runbook before any fold or dispense, so the run can never advance while the contract is unavailable. A missing runbook is re-rendered only by the nx version that created the run (published atomically via a random-suffix temp and rename, mirroring run.json's writes); with a different creator version the response is an exit-0 refusal that leaves git history and run state untouched. Terminal responses stand alone: completed runs, and active runs whose steps are all terminal, emit their completion without requiring the runbook. The stored-bytes read goes through a no-follow, non-blocking descriptor re-validated by inode identity, so a swapped-in symlink, FIFO, or replacement file fails the invocation instead of being followed, blocking, or masquerading as the contract.
A migration recorded into an orchestrated run now matches the classic loop's execution semantics. skipAgentic is honored: a hybrid's waived prompt or a generator's waived validation logs the waiver and the step succeeds instead of parking. Generator output is captured whenever an agent step may consume it. A generator-only migration that made changes parks for a validation pass when the run validates: the worker installs what the generator changed, emits a kind: generator-validation block, and records the step's awaiting kind, deferring the commit to the fold so a failed validation leaves the changes uncommitted for review. The await dispense words a validation step as validation and offers no skipped outcome; a skipped handoff on one still folds as completed, since the generator's changes are already applied. The marker transition persists what a retry cannot recompute without rerunning the generator: the waiver, the owed validation, and whether any files changed. Retries re-install from the step's baseline before handing work back and re-emit exactly the agent work that is owed, and a retried no-op neither parks for validation nor builds a commit that would absorb unrelated pending diffs under its name. Recorded workers always emit the structured block; ambient agent detection no longer decides the protocol surface.
A recorded migrate step that hands work back to the driving agent (the prompt half of a hybrid migration, or a validation pass over a generator's changes) now defers its commit to the reconcile fold, so each migration lands as one commit and a failed hand-back leaves its changes uncommitted for review, matching the classic loop. A skipped handoff over applied generator changes folds as completed so those changes cannot be stranded as debt; a no-op hybrid's skipped prompt still folds as skipped, and its dispense keeps offering the skipped outcome. The worker also stores each parked step's block payload under the run directory (agent-work/<step>-attempt-<n>.json, atomically, failing the attempt when the store fails), and the reconcile dispense re-emits it, so a session that lost the original block to a compaction or restart gets the work restated instead of a pointer into stdout it no longer has. When no stored copy is usable the dispense synthesizes the payload from the plan's prompt path or the tree-pointing validation marker; a stored prompt payload must name exactly the plan's instructions file. Retained-generator retries re-hand the newest surviving copy inside the lineage boundary recorded with the generator marker (generatorCompletedAtAttempt), failing closed when the boundary is absent; a reset-backed retry invalidates the superseded copies and a terminal fold removes them. Lookup and cleanup enumerate the stored files, so persisted counters cannot drive unbounded scans, and the run state validates both counters as the integers nx writes.
…e run Agent sessions driving an orchestrated migrate run can report issues through the handoff file (issues / issueUpdates). nx validates each report, mints ids and fingerprints, maps named migrations to plan steps, and tracks dispositions (recorded / resolved / deferred-final) with resolver credit, commit fences, and claims in run.json. Dispensed steps carry a bounded digest of known issues; recorded issues are claimed for in-scope steps; landed commits associate the issues they carry; retry-clean reopens the discarded attempt's resolutions. Full detail is archived under the run's issues directory, identity-checked and reconstructed with a warning when lost; content authenticity of the archive is an explicit non-goal (run.json stays authoritative). The run-state format bumps to v2 so an older nx refuses issue-bearing runs instead of silently violating the ledger rules. Dark behind NX_MIGRATE_ORCHESTRATOR.
An agent driving an orchestrated run can loop: re-invoking reconcile without acting on its response, or retrying the same failing step forever. Neither loop had a circuit breaker. No-progress escalation: the orchestrator persists a response streak on the run (the noProgress field), fingerprinted as a digest of the whole persisted state minus the streak itself, so any durable transition (a step transition, a generator marker, an issue settle or claim, a ledger append) resets the count. After 3 responses with no durable transition between them, the response is emitted as a 'no-progress' action that names the loop and keeps the step's own instructions and commands, so acting on them stays possible. Never a step status: nothing durable changed, and a status would have to be walked back once the agent acts. A still-running worker proven young by a parseable start time is exempt until the 15-minute hang threshold; a running step that cannot prove its age counts like any repeated response. The accounting is write-before- output at-least-once: a crash between the write and the emission costs at most a count one ahead, and the escalation is advisory. Rearm cap: once a step has consumed 3 rearms, its failed/died dispense adds escalation guidance (fix the cause, choose a non-retry option, or ask the user) and withholds the preselected retry 'next', since each rearm resets the response streak and a blindly-followed retry 'next' would loop past every escalation. An explicit --step-action=retry stays honored where the state machine otherwise allows it; the cap never refuses. The runbook documents the no-progress action in its loop discipline.
… scenarios Init emits the runbook before an initialized block and every later dispense is driven through the reconcile command, so the previous scenarios no longer matched the contract. Nine scenarios now drive a fake agent through the run: the base loop with validation, prompt and hybrid parks and a mid-park resume; a waived hybrid and a skipped prompt; a rejected handoff and a failed one retried; no-progress escalation and reset; issue routing, digest caps and commit issue ids; a killed worker recovered by retry-clean and by adopt; and plan-mismatch refusal with the accepted-init gitignore fallback and a same-plan resume.
The spec replaced RUNBOOK.md with unlink plus create between the lstat and the open. ext4 hands the freed inode number to the next create in the same directory, so on Linux the replacement matched the inspected dev/ino and the guard let the read through. Rename a sibling file over the path instead: both files exist at once, so the inode always differs. The guard comment overclaimed identity against any replacement; it now states the case it covers, a followed symlink on Windows.
…t v1 - Read handoff files through the same O_NOFOLLOW descriptor and inode check the runbook read uses, so a symlinked handoff is rejected instead of followed; the helper now lives in agentic/handoff.ts and both reads share it. - Reject an "outcome" handoff value other than "skipped" instead of parsing it as absent. - Keep the run-state format at v1: the orchestrator ships dark, nothing reads v1 runs yet, and the ledger fields are additive and read-validated. Drop the lazy restamp and the bump-rule clause that justified it. - Fix the remaining-work enumeration in the retry prompts (a validation pass is also a handed-back half), state that the debt check in canOfferCleanRetry is run-wide on purpose, and make the updateRunState contract match its out-param callers. - Runbook: the dispensed reconcile, worker and next commands are the exception to "do not run nx migrate". - Point the cross-version missing-runbook message at runs created before runbooks existed, and name the two consumers of fragments.ts.
The handoff read guarded only the final path component, and the removal on retry or fold used a plain rmSync. An agent scoped to the handoffs dir could replace a package segment, or the handoffs dir itself, with a symlink and have nx read back or delete a file of the handoff's fixed name anywhere on disk. Checking the parent chain before the read or the unlink cannot be atomic in Node, so remove the parents instead: handoffs now sit directly inside the handoffs dir, named by the step id for orchestrated runs and by scope+package+name for the per-step runner, with "+" (which npm rejects in package names) stripped from migration names. The read and the removal refuse a handoffs dir that is not a real directory, and every handoff read, the per-step runner's included, goes through that check.
The per-step runner named a handoff after the sanitized package and migration name. The sanitizer folds many characters to `_`, so distinct migrations (`a+b`, `a_b`, `a/b`) mapped to one file, and the classic flow keeps consumed handoffs, so the second migration read the first one's handoff and recorded success without running. A 250-character name also produced a basename past the per-component limit. Name the file with a 64-character sanitized prefix plus a SHA-256 of the raw package and name pair, so the name stays readable, bounded and collision-resistant.
The readable prefix was cut at 64 UTF-16 code units, so a name made of multibyte characters still produced a basename past the 255-byte per-component limit, and the cut could split a surrogate pair. Cut on code points against a 64-byte UTF-8 budget instead.
Apply the comment audit on the migrate orchestrator changeset: drop 91 comments that restate the code, a neighbor, or a test's own name and assertions, and shorten 207 to the facts the code cannot give. Replace the unicode ellipsis and em dashes added to the agent prompt strings with ASCII; the spec regexes that asserted the ellipsis follow.
…tep [Self-Healing CI Rerun]
Contributor
There was a problem hiding this comment.
Nx Cloud has identified a flaky task in your failed CI:
🔂 Since the failure was identified as flaky, we triggered a CI rerun by adding an empty commit to this branch.
🔔 Heads up, your workspace has pending recommendations ↗ to auto-apply fixes for similar failures.
🎓 Learn more about Self-Healing CI on nx.dev
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Current Behavior
Only with
NX_MIGRATE_ORCHESTRATOR=true, which is off by default.When an AI agent runs
nx migrate --run-migrations, nx records the run and hands the agent one migration at a time. The agent is never told how the loop works: the first thing it sees is already a migration to run. If its session gets compacted, crashes, or restarts, nothing on disk tells it how to pick the run back up.Migrations run this way differ from the normal
nx migrateloop:skipAgenticis ignored, so the migration still stops and waits for the agent.The agent has no way to tell nx about problems it found. Nothing stops an agent that keeps asking for the next step without doing anything, or keeps retrying a failing step.
Expected Behavior
Still behind
NX_MIGRATE_ORCHESTRATOR=true.The first response is a runbook that explains the loop, plus one command to get the first migration. Resuming a run returns the same runbook, so a restarted session always finds its way back.
Recorded migrations behave like the normal loop:
skipAgenticis honored.The agent can report problems in its handoff file. nx keeps the list and shows the relevant ones with each following migration. It records which commits fixed them.
After three responses with nothing changed, or three retries of one step, nx tells the agent to change approach instead of looping.
Related Issue(s)
NXC-4627
Implementation Notes
--agenticloop feeds its per-step agents today:prettier --write --ignore-unknownoroxfmt --no-error-on-unmatched-pattern, resolved by nx on every dispense and prefixed with the package manager exec command; a workspace with no formatter is told not to run one, and the rule names the exact commands for a migration that itself changes the formatter) instead ofnx format:writenx format:writejoins the list ofnxcommands the agent must not runnx affected --files=<changed paths> -t ...instead of a barenx affected -t ...nx show project <name> --jsonis named as the source of truth for inferred targetsRUNBOOK.mdis written beforerun.json, so a run is never visible without it. Resume re-emits the stored file, so the instructions stay tied to the nx version that started the run. A missing file is rewritten only by that version; any other version exits 0 without changing anything.--step-action=retrystill works.handoffsdir (handoffs/<step id>.json; the normal--agenticloop useshandoffs/<scope>+<package>+<name>-<sha256 of the package and name pair>.json, since the sanitized prefix alone is ambiguous and unbounded), so no directory an agent may write lies between that dir and the file. nx reads or removes a handoff only whenhandoffsis a real directory; a symlink in its place is reported as an unreadable handoff and never followed or deleted through. Checking nested parents before a read or unlink cannot be made atomic in Node, which is why the layout changed instead.--validatehelp text, and the check that real agents follow the runbook (NXC-4817)View Polygraph session ↗