Fail nightly fast on env drift vs fvdb-core - #319
Open
harrism wants to merge 2 commits into
Open
Conversation
The nightly builds the fvdb-core wheel using fvdb-core's env/build_environment.yml and installs it into the benchmark env defined in this repo. When those pins disagree the wheel is compiled against one libtorch and loaded against another, and the run dies at `import fvdb` with an undefined-symbol ImportError -- most recently 2026-08-20, when fvdb-core moved to PyTorch 2.13 while this repo still pinned 2.11. That failure is both late and expensive: it surfaces only after the GPU runners are provisioned, and in the comparative job only after roughly an hour of build time. Add a check-env-drift job that compares the pytorch-gpu, cuda-version and python pins and fails on ubuntu-latest before any EC2 instance is started. start-build-runner now also needs check-env-drift. Every runner-provisioning job descends from it, so gating that one job blocks the whole EC2 chain. Two implementation notes: - The fvdb-core environment is fetched at the exact commit the run builds from rather than read off disk. The build job clones fvdb-core only when no cached wheel artifact exists, so an on-disk check would silently skip on cache hits. - An unreadable pin fails the job. Without that, renaming or reformatting a pin line would compare empty to empty and pass silently. This is the third occurrence of this drift (1b6956f 2.8->2.10, 6c2ede6 2.10->2.11, and the 2.11->2.13 break above). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Mark Harris <mharris@nvidia.com>
harrism
requested review from
matthewdcong and
swahtz
and removed request for
a team
August 21, 2026 04:33
Codex review found that check-env-drift checked out the workflow's triggering
ref, while the build and benchmark jobs check out
${{ github.event.inputs.branch || 'main' }}.
For the scheduled run those are the same, so the gate behaved correctly. For a
workflow_dispatch run against a branch they diverge: the gate would validate
main's benchmark_environment.yml, pass, and let both EC2 chains start, while
the build and benchmarks used the dispatched branch's pins. That is exactly
the false-negative this job exists to prevent, on the one code path where a
human is most likely testing a pin change.
Pin the gate's checkout to the same expression. Verified afterwards that all
five runner-provisioning jobs remain transitively gated and that the always()
stop-runner jobs are unchanged, so no EC2 instance can leak.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Mark Harris <mharris@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The nightly builds the fvdb-core wheel using fvdb-core's
env/build_environment.yml, then installs it into the benchmark env defined in this repo. When those pins disagree, the wheel is compiled against one libtorch and loaded against another, and the run dies atimport fvdb:This has now happened three times — 1b6956f (2.8→2.10), 6c2ede6 (2.10→2.11), and the current 2.11→2.13 break (run 32362276082, fixed by #318).
The failure is both late and expensive: it only appears after the GPU runners are provisioned, and in the comparative job only after ~1h of GSplat build time. It also reports under the misleading step name "Download mipnerf360 dataset".
Change
Add a
check-env-driftjob that compares thepytorch-gpu,cuda-versionandpythonpins between the two environments and fails onubuntu-latestin seconds.start-build-runnernow alsoneeds: check-env-drift. Every runner-provisioning job descends from that one, so gating it blocks the entire EC2 chain — verified:start-build-runnerstart-benchmarks-gpu-runnerstart-comparative-gpu-runnerfvdb-reality-capture-unit-benchmarksfvdb-reality-capture-comparative-benchmarksTwo implementation details worth review
The fvdb-core env is fetched at the pinned SHA, not read off disk. The obvious placement — after the build job's
Clone fvdb-corestep — is wrong, because that step is conditional:On days when the cached wheel artifact hits, fvdb-core is never cloned and an on-disk check would silently skip. Fetching
raw.githubusercontent.com/openvdb/fvdb-core/${FVDB_CORE_SHA}/env/build_environment.ymlis cache-path independent and pins to the exact commit the wheel is built from.An unreadable pin fails the job. Without the emptiness check, renaming or reformatting a pin line would make both
grep -oPresults empty, compare equal, and pass silently — a guard that quietly stops guarding.Testing
Dry-ran the comparison logic against the real files on both sides:
and against a simulated post-#318 benchmark env, where all three keys match and the job exits 0.
Sequencing
This check will fail on
mainuntil #318 (sync benchmark env to PyTorch 2.13) merges, since that drift is real and present right now. Merge #318 first.A companion PR adds the mirror-image check to fvdb-core PRs.
🤖 Generated with Claude Code