feat(core): let an orchestrated migrate run give up on migrations, adopt hand-applied ones, and cap retries - #36986
Draft
leosvelperez wants to merge 12 commits into
Draft
feat(core): let an orchestrated migrate run give up on migrations, adopt hand-applied ones, and cap retries#36986leosvelperez wants to merge 12 commits into
leosvelperez wants to merge 12 commits into
Conversation
Contributor
|
View your CI Pipeline Execution ↗ for commit 450f46e
☁️ Nx Cloud last updated this comment at |
leosvelperez
changed the title
leosvelperez
added this pull request to stack #36919
September 10, 2026 08:11
✅ Deploy Preview for nx-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Deploy Preview for nx-dev ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
![nx-cloud[bot]](https://avatars.githubusercontent.com/in/80458?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
leosvelperez
force-pushed
the
nxc-4819
branch
from
September 10, 2026 09:18
040d7c0 to
2b72e48
Compare
leosvelperez
force-pushed
the
nxc-4819
branch
from
September 10, 2026 15:18
5ebc011 to
1778d90
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
…ed step A failed or died step could only be retried, skipped, or (from died) adopted, so a migration nobody could fix had to be skipped, which reads as "not applicable" in the run's record. The new 'unresolved' status is terminal and keeps the attempt count and the last failure on the step; '--step-action=unresolved' reaches it from failed or died. The tree is handled like a skip: the install the dead worker owed runs, and on a committing run the dirty tree is recorded as commit debt. Adopted steps carry an 'adopted' marker so the completion report can list them without parsing summaries. No run-state format bump: the orchestrator is dark behind NX_MIGRATE_ORCHESTRATOR.
…p is given up on Giving up on a step left its tree unhandled beyond the install a skip owes. Now the reconcile decides what happens to it. When the generator never completed and the same restore point a clean retry needs exists, the agent resets the tree and nx accepts only a verifiably clean one, reopening the issue resolutions the reset discards. Otherwise a committing run commits the partial result under the migration's name with an "(unresolved)" marker, through the parent session's broker when there is one, so a later revert does not have to untangle it from the next step's work; a run without commits keeps the tree as a skip does. The step's transition, a run issue carrying its last failure to the completion report, and the step's reference to that issue land in one locked write. The issue is unscoped and deferred so no later step claims it; an agent report with the same text is taken over rather than merged. A lost archive file warns and never undoes the transition. The broker names a marked commit as its own request, so giving up after a worker's commit failed at its install lands a commit instead of replaying that failure.
…ed by hand A migration whose generator failed and was then applied by hand had no way into the run's record except a retry that reran the generator. The adopt action now accepts a failed step as it does a died one: the tree is taken as the migration's result, the step succeeds with the adopted marker, and a committing run commits it under the migration's name. Through the parent session's broker a failed step's adopt is its own request. The worker's own commit request for that attempt ended in the failed install the step recorded, so replaying that answer, as a died step's adopt does on purpose, would skip the install and the commit.
…earms The rearm cap was advisory: past three rearms the failed or died dispense argued against retrying and withheld the preselected continuation, but an explicit retry was still accepted, so a step could be retried forever. The cap is now two rearms (three attempts) and it refuses: the reconcile rejects retry and retry-clean past it, naming the cap, and the dispense lists only adopt, skip and unresolved.
…d migrate run leaves work unresolved The completion report counted applied and skipped steps and the master session exited 0 for any completed run, so a migration given up on and a problem nobody fixed both read as success. One tally now classifies every step: applied, adopted, skipped and unresolved, with each given-up migration listed with the failure it was given up on. The orchestrator's completion block, the resume banner and the master session's report share it, and the master session exits 1 when a step is unresolved or a reported issue is still unresolved. The run itself stays completed. Failure text is agent or generator output printed verbatim, so the report collapses it to one line: a break inside it could open a forged block.
…and the retry budget The runbook described a failed or died step as a choice between retry and skip, and the dispense said nothing about how many retries a step had. The agent now gets the decision rule: diagnose the failure first, retry only with a plausible fix in hand, two retries per step with the user asked before the last one, adopt for a result already in the tree, skip for a migration that does not apply, and unresolved to give the migration up. With nobody to answer, it gives the step up and continues; an unfinishable prompt reaches that option by writing the failed handoff itself. Each failed and died dispense opens with the retries left. The exit code is promised only for a session nx started itself; an agent driving the loop reads the completion block's unresolved tally instead.
…grate orchestrator e2e Five orchestrator cases: give up on a killed worker by reset and check the minted issue, commit the partial tree of a given-up prompt step under its name marked unresolved, adopt a failed prompt step applied by hand, refuse a retry past two rearms, and a master session that exits 1 with the given-up migration in its report. The fake agent gives a step up through the dispense's unresolved option only when FAKE_AGENT_FAIL_PROMPTS makes its prompt-based migrations fail; generator validation handoffs still succeed and retry-failed still retries otherwise.
…adopted, skipped or given up A worker commits its migration before it reports the outcome, so a step can fail or die with its commit already in the ledger. Adopting such a died step re-published the worker's own commit request; under the session broker the cached answer came back and the ledger entry landed twice, and in-process the tree was committed again under the worker's name. The failed and died dispenses also offered skip and unresolved for the step, and acceptance took them, so the run could record a migration whose result sits in history as skipped or given up. Adopt now asks under its own commit-adopt request whenever the step is failed or its commit already landed, so a cached worker answer is never replayed and repairs made since land as their own commit. Skip and unresolved are withheld from both dispenses and refused at acceptance, naming the landed sha and pointing to adopt; the retry budget, cap and retry-clean fallback lines, the runbook and the multi-repo migrate skill say the same. The persisted failed-adopt summary no longer claims the tree was applied by hand. The overlong-id digest test gives both steps the same attempt and failure so only the digest tells the summaries apart; the broker seam comment is scoped to a failure with no landed entry.
…ted migrate step is given up A died step carries no outcome (markDied records the status only and a re-arm rebuilds the step without one), so giving it up reached the completion report and the minted issue with "no failure detail was recorded". The died -> unresolved transition now records the death as the step's failure, and the issue is minted from the transitioned step instead of the pre-transition snapshot. Also: the mintUnresolvedIssue doc no longer claims the migration id keeps two steps from folding, the archive-shape comment sits above newIssueArchiveRecord again, and the commitForStep contract names failed adoption.
… out of skip and unresolved The gate that withholds and refuses skip and unresolved for a landed commit reused the reset-hazard check, which ignores an earlier attempt's commit once it is an ancestor of the re-captured ref and cannot see a commit the session landed for the current attempt when the worker died between the broker's answer and its ledger append. A retried step that failed again, or a worker killed in that window, could be recorded skipped or unresolved with its commit in history. The gate, both dispenses and the adopt request choice now use the last ledger entry covering the step, whichever attempt landed it. Reconcile recovers a dead worker's brokered commit from the session's cached answer before validating any step action, and appends it once: landed entries record the attempt that landed them (ownerAttempt), with the sha as the key for entries written before the field existed. A commit a worker made in-process and died before recording leaves no answer and is not detected; the runbook, the repo skill and the PR body say the options are withheld for a commit the run has on record.
…ases in migrate orchestrator specs The closed-set JSDoc on NewerRunStateFormatError demanded a format bump for every new member, but `unresolved` joined the step statuses at v1 on purpose: runs are not supported across Nx versions while the orchestrator is dark, and the bump waits for its promotion. The `adopted` field comment excluded a dead worker's landed commit, which adopt also accepts. Specs gain the `recorded` disposition in the master-session exit table and a died step with no cached broker answer, which keeps skip and unresolved open and records no entry.
leosvelperez
force-pushed
the
nxc-4819
branch
from
September 10, 2026 19:38
e75eeee to
450f46e
Compare
…ases in migrate orchestrator specs [Self-Healing CI Rerun]
Contributor
There was a problem hiding this comment.
Nx Cloud has identified a flaky task in your failed CI:
🔂 Since the failure was identified as flaky, we triggered a CI rerun by adding an empty commit to this branch.
🔔 Heads up, your workspace has pending recommendations ↗ to auto-apply fixes for similar failures.
🎓 Learn more about Self-Healing CI on nx.dev
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Current Behavior
This only applies to orchestrated migrate runs, behind
NX_MIGRATE_ORCHESTRATOR.When a migration fails, nx offers the agent three choices: retry, retry over a clean tree, or skip. A dead worker gets a fourth choice, adopt, which keeps the working tree as the result. Nothing lets the agent say "this migration is not going to work, move on". Nothing lets it keep a fix it applied by hand after a plain failure. A skip pretends the migration did not apply to the workspace, which is wrong for both cases.
The agent can retry a failing migration forever. Each retry costs time and the failure is usually the same.
At the end, the run reports success even when a migration never landed. The user has no list of what was given up, and a session that nx started for the run exits 0.
Expected Behavior
After a failure or death the menu gains:
adoptafter a plain failure: the tree as left is the result, committed under the migration's name.unresolved: the agent gives the migration up and nx files the failure as a run issue. Safely discardable changes are reset; others stay and, with commits on, land under the migration's name marked unresolved.A migration with a commit on record gets adopt instead of skip and unresolved, committing what changed since. Nx recovers a commit the session landed for a worker that died before recording it. A commit made in-process, or one whose session ended first, is not detected. A retry resumes the remaining work without reapplying the generator.
nx refuses third retries.
The completion report counts applied, adopted, skipped and unresolved migrations and lists each given-up one's failure. A session nx started exits 1 on a given-up migration or open issue.
Related Issue(s)
NXC-4819
Implementation Notes
unresolvedis a terminal step status, so a resumed run never re-dispenses a given-up migration.