feat(core): select affected tasks instead of affected projects - #36841
Draft
AgentEnder wants to merge 14 commits into
Draft
feat(core): select affected tasks instead of affected projects#36841AgentEnder wants to merge 14 commits into
AgentEnder wants to merge 14 commits into
Conversation
AgentEnder
force-pushed
the
feat/nxc-4859-affected-task-granularity
branch
from
August 28, 2026 20:01
1340f19 to
be5c9d6
Compare
Contributor
|
View your CI Pipeline Execution ↗ for commit aa507c2
☁️ Nx Cloud last updated this comment at |
AgentEnder
changed the base branch from
feat/nxc-4861-files-input
to
feat/nxc-4859-task-based-affected
August 28, 2026 20:22
AgentEnder
force-pushed
the
feat/nxc-4859-affected-task-granularity
branch
from
August 29, 2026 17:06
be5c9d6 to
cf266fc
Compare
Tests the changed paths against each hash-plan instruction's globs, rather than resolving instructions to file lists and intersecting. A deleted file has no entry in the workspace file map, so a resolved list could never contain it and every rename would be missed. It also inverts the cost to O(unique instructions x changed files), and instructions are interned so a glob set shared by a thousand tasks is compiled once. TaskOutput is deliberately skipped: it resolves to a dependent task's build artifacts, which are gitignored and do not exist yet when affected runs. Dependency changes reach a consumer through task-edge propagation instead.
Builds the full candidate task graph, plans it, matches changed paths against each instruction's globs, then propagates along dependentTasksOutputFiles edges with three bits per task over a topological order. The locators still run and seed the result. A file intersection alone under-selects three ways they cover: ProjectConfiguration resolves to no files, so a project.json edit can be invisible; lockfile and external-dependency changes are not paths; and a blanket trigger like nx.json has no fileset to match. Seeding keeps this a superset of the project-grained answer. Extracts the graph marshal and the locator fan-out so filterAffected and computeAffectedTasks share one locator path rather than two. NOT yet wired to any command: planning every candidate task costs ~3ms/task on this repo, so discovering that 4 tasks are affected costs ~760ms of planning across 171 candidates. See the notes on the PR before enabling this anywhere.
Adds pruneTaskGraphToSelection, threads an optional TaskSelection through runCommand so the pruned graph reaches the runner, and dispatches on nx.json affected.granularity (NX_AFFECTED_GRANULARITY overrides). Default stays project-grained. The prune keeps each selected task's dependency closure: an affected task's upstream still has to run or restore from cache. It is applied before validation so a cycle or atomizer error names what will actually run, and re-applied after sync generators rebuild the graph. Gate only engages for `nx affected` with a target; `nx graph --affected` has no target to select against and stays project-grained.
A task can only be affected if a changed file reaches its plan, and every
route there marks the owning project: its own files, a dependency's files
inlined by ^ inputs (whose dependents the reverse walk picks up), and
{workspaceRoot} filesets, which getImplicitlyTouchedProjects attributes. So
projects-of(affected tasks) is a subset of the project-grained set, and
planning anything outside it is wasted.
Measured on this repo, selection plus hashing the selected set:
a README change 1369ms -> 6ms
a leaf e2e spec 1956ms -> 523ms
Planning every candidate to discover that nothing is affected was the whole
cost of the docs-only case, which is a common CI shape.
Also shares one HashPlanner per project graph, so the candidate pass populates
subtree_memo and instruction_pool once rather than per call.
Two target-configuration shapes create a real dependency the project graph
never records, so the reverse walk in filterAffected cannot see either:
inputs: [{ input: 'production', projects: ['shared-config'] }]
gather_project_inputs resolves the list and inlines those projects'
filesets into the task's plan, so its hash genuinely depends on them.
dependsOn: [{ projects: ['api'], target: 'build' }]
processTasksForMultipleProjects builds a task edge to a named project.
Project-grained affected under-selects for both today: a change to a project
referenced only this way selects nothing. Task granularity has to close the
hole, because it uses the project-grained answer to bound which tasks are
worth planning.
Also narrows the seeds. The file matcher is precise for ordinary source
files, so seeding every task of a touched project there just reproduced
project granularity. Seeds now cover only what glob matching cannot see:
project.json/package.json edits, lockfile changes, and the deleted-project
blanket.
Task-grained affected plans the candidate tasks to decide what is affected, and the hasher then plans the survivors. A planner carries subtree_memo, instruction_pool and external_deps_mapped across getPlans calls, so handing the same instance to both makes the second pass mostly memo hits. Threaded as an argument from computeAffectedTasks through runCommand to createTaskHasher, rather than held in a module-level cache keyed on the graph. A cache works but retains napi objects for as long as the graph is reachable; passing it explicitly scopes the lifetime to the command and avoids depending on GC timing, which a WeakRef would not. Measured on this repo, selection plus hashing the selected set, paired runs: a spec-only change 3699ms -> 3118ms a source file 3987ms -> 3647ms a project.json 3788ms -> 3260ms
Drops the nx.json affected.granularity key. This is a trial flag that we expect to remove once task granularity becomes the default, and a config key would need a migration to take back out.
I/O tracing turns an observed read of a generated artifact into a FileSetInput with includeIgnored, which can preclude an explicit dependentTasksOutputFiles input. The plan then holds a disk-backed Files instruction reading a dependency's build output with no TaskOutput anywhere in it, and propagation missed the edge entirely. dependentOutputEdges now builds consumer -> producer edges from both sources: an explicit TaskOutput, whose producer id is recovered by matching the embedded outputs, and an includeIgnored fileset whose pattern overlaps a producer's declared outputs. Matching is pattern-to-pattern against declared outputs, never the filesystem. The artifact is the thing the run would produce, so it is absent from disk and gitignored, and any check that consults disk answers "no edge" for every task that has not been built. Overlap compares literal prefixes segment-wise, so dist/libs/ui does not swallow dist/libs/ui-legacy. Producers are searched over the whole dependency closure. TaskOutput does not record whether transitive was set and a traced read cannot say how deep the producer sits, so a narrower scope could miss an edge; over-reporting costs a cache hit, missing one skips a task that needed to run. TaskOutput stays out of direct changed-file matching, since its artifacts can never appear in a diff.
The first cut walked the dependency closure per consumer, cloning task ids at every hop and re-reducing the same glob set once per task that referenced it. That was 216ms for 335 tasks here, and it scales with tasks x closure. Instructions are interned, so a glob set shared by a thousand tasks has one id and is now reduced to its literal prefixes once. The closure walk moves over positions into a flattened graph rather than hashing ids, reuses one generation-stamped scratch buffer per rayon worker, and runs in parallel. Same 1560 edges, 216ms -> 54ms. Also returns a consumer -> producers map instead of an edge list, which halves the strings crossing the boundary and is the shape the caller builds anyway.
The blanket rule fired whenever the project-grained pass happened to select every project, as a proxy for "a blanket locator returned everything". In a small workspace an ordinary source change does exactly that, so every task of every project got seeded and task granularity collapsed back to project granularity. Only a deleted project config actually needs the widening: its project is gone from the graph, so no surviving task has a fileset that names it. The check mirrors projects_from_project_glob_changes, which decides the same thing by asking whether the file is still on disk. nx.json needs no seed at all. Every plan carries it in the always-on workspace fileset (ALWAYS_ON_FILES in hash_planner), so the matcher already reaches every task precisely.
…locators The project locators and the task matcher both answer "which project owns this path", and both carried their own root map plus the same comment explaining why create_project_root_mappings cannot be used. Fixing that shared helper later now means finding one workaround rather than two. ProjectRoots and normalize_path move to affected/project_paths.rs with tests for the cases that motivated them: an empty project root, a whole-segment match so libs/a does not claim libs/a-legacy, and Windows separators. Also trims the matcher while it is being touched. Only instructions that matched a file are kept, so task selection is membership rather than collecting every hit; the per-task index vector is now built only under collectMatches, the reduce runs on rayon, and file owners borrow from the graph instead of allocating a String each. 125ms -> 42ms for 335 tasks.
…ing twice Task-grained selection plans every candidate task, then the hasher planned the surviving ones all over again. Planning costs about 3ms per task on this repo, so the second pass was roughly a second on a 335 task graph, and it is what put the task path 18% behind the project-grained one. subsetHashPlans narrows an existing HashPlans to a task list. A plan is a vector of interned instruction ids, so this filters a map and clones an Arc; the work that made the plans is shared, not repeated. It returns null when any requested task has no plan, which is the caller's signal that the plans describe some other task graph. The hasher then plans for real. That happens with an I/O snapshot bundle, whose plans are fetched after selection and so describe different hashing, and after makeAcyclic, which drops edges and changes which upstream outputs a task reads. Verified hash-identical across all 276 tasks of a real affected run, reused against freshly planned. Warm cache totals, median of three, this repo: changed task before task after project utils/path.ts 7449ms 6315ms 6291ms ab-testing.spec.ts 6510ms 5102ms 6374ms The wide case reaches parity, and the narrow case is 20% ahead because it also hashes 233 tasks rather than 278.
The env-vars-documented conformance rule fails on any NX_ variable read in source that has no row in the reference table, and NX_AFFECTED_GRANULARITY had none. That was the only violation in the run; every other rule passed.
oxlint rejects no-duplicate-imports, and both files had picked up a second import from a module they already imported: subsetHashPlans from ../native in the hasher, and TargetDependencies from ../../config/nx-json in affected.
AgentEnder
force-pushed
the
feat/nxc-4859-affected-task-granularity
branch
from
August 30, 2026 02:25
1487e21 to
aa507c2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Current Behavior
nx affectedselects projects, so every target of every dependent runs whether or not the change can reach it. Changing a spec file inpackages/nxselects all 45testtasks here, including projects whose test inputs never see that file.Two target configs also create a dependency the project graph never records:
{ "inputs": [{ "input": "production", "projects": ["shared-config"] }], "dependsOn": [{ "projects": ["api"], "target": "build" }], }gather_project_inputsinlines the named project's filesets into the consumer's plan, andprocessTasksForMultipleProjectsbuilds a task edge to it. Neither needs a project-graph edge, so the reverse walk misses both, and a change to a project referenced only this way selects nothing.Expected Behavior
nx affectedselects tasks whose inputs the change reaches. We build the candidate task graph, resolve each task's hash plan, and match the changed paths against each instruction's globs.Off by default, opt in with
NX_AFFECTED_GRANULARITY=task. Env var only, so the trial leaves no config to migrate away later.Selection
Nx repo, changing one spec file. Selection marks 74 of 280 candidate tasks: 65
lint, 2test, 1build, and nothing inbuild-baseortypecheck.-t test, test tasks-t test, whole graph-t build-t build,test,lintThe first row is the win. The rest of each graph is the dependency closure of the selected tasks, which has to stay so the selected ones can run; those restore from cache.
lintreads nearly every file in a project, so a graph dominated bylintre-inflates and-t build,test,lintbarely moves.Cost
Selection now plans the candidate tasks, which the hasher then reuses instead of planning again, so a run plans once rather than twice.
show projects --affectedaffected -t lint, fully cachedTask hashes are unchanged: this branch, master and nx 23.1.2 all hit the same cache entries.
show projects --affectedis still project-grained, so there is no task column for it.Related Issue(s)
NXC-4859