chore: make the documented verification commands work - #388
Draft
gblanc-1a wants to merge 3 commits into
Draft
Conversation
added 3 commits
August 6, 2026 10:55
Every command below was run to confirm it succeeds, because the previous set contained instructions that fail: - `pnpm run lint:fix` does not exist at the root, and `npx eslint <path>` cannot run there either (no root eslint.config.mjs). Replaced with the per-package and filtered forms. - `pnpm -C packages -r <script>` looks scoped to packages/ but resolves the whole workspace, so it also runs the extension, lib/, website/ and github-actions/: 4920 lines of output where `--filter './packages/*'` gives 181. Added a Verification section for three traps that let a broken change look verified: - Piping a test or lint run through grep/tail replaces the runner's exit code with the pipe's, so a failing suite reports success. - `test:unit` executes test-dist/ and compiles nothing (`pretest` belongs to `test`), so it silently re-runs the previous build. Confirmed by appending an always-throwing test to a compiled test file: the whole suite still passed. - Lint and tsc resolve @ai-primitives-hub/* through built dist/, so a stale build reports errors absent from the checked-out source. On a clean main checkout this produced six phantom errors that disappeared after a rebuild, with no source change. Also ignore .gitnexus/, a local code-intelligence cache of ~111 MB of single-line JSON that search tools should never read.
The flow needed six commands in a specific order, and two of the obvious ways to run them are wrong: `pnpm -C packages -r <script>` resolves the whole workspace, and `test:unit` executes a stale test-dist/. Both are now covered by scripts. New root scripts: - build:packages / test:packages / lint:packages — filter on @ai-primitives-hub/*, the same filter CI uses. 181 lines of output where the recursive form gives 4920. - lint:extension, and lint:fix = packages + extension. AGENTS.md has been telling everyone to run `pnpm run lint:fix` at the root, which did not exist. - test:extension = compile-tests && test:unit, so unit tests cannot pass against a build that predates your edit. - verify:packages (~30s) and verify (~110s), which chain the whole thing in dependency order and give one exit code. `test:unit` stays a separate script and keeps its exact definition: CI applies post-compilation fixes between compile-tests and test:unit, so a pretest hook would re-run the compile after those fixes. Every pre-existing script (build, watch, lint, test, compile, compile-tests, test:unit, package:vsix) is byte-identical, so no workflow changes. Also set noWarnOnMultipleProjects on the shared eslint resolver config. Every package legitimately has a src and a test tsconfig, so each lint run printed a "Multiple projects found" warning to stderr that reads like a fault. AGENTS.md, packages/AGENTS.md, and the two extension guides now document these scripts, plus the redirect-and-tail idiom for reading a failing verify without pulling 5000 lines into context.
The first cut of `verify` stopped at unit tests, which excluded test-dist/test/suite/** — the only layer that exercises command registration, package.json contributions and real activation. A change to either would have passed `verify` unnoticed. Two corrections: - `test:extension` now delegates to the extension's existing `test:all` (compile-tests && test:unit && test:integration) instead of hand-rolling compile-tests && test:unit. The composite already existed; duplicating it minus integration was both redundant and less complete. - `test:extension:unit` keeps the fast path (~20s, no VS Code) for the inner loop. `verify` is now ~132s and runs 2196 unit + 7 integration tests. The integration layer launches a real VS Code through @vscode/test-electron and needs a display; CI already wraps it in xvfb-run on Linux. The guides now state which layer sees what, so "unit tests pass" is not mistaken for "the command is registered".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Working through the auth-diagnostics change surfaced several places where the documented workflow either fails outright or reports success on work it never checked. This fixes the commands, adds scripts so the correct sequence is one invocation, and records the traps in the
AGENTS.mdguides.Three of the findings are correctness issues, not conveniences — each one lets a broken change look verified:
grep/tailreplaces the runner's exit code with the pipe'stailexits 0; bare, it exits 1. A Mocha run reported exit 0 while printing2 failing.test:unitexecutestest-dist/and compiles nothingtscresolve@ai-primitives-hub/*through builtdist/maincheckout with a stale build,packages/infralint reported 6 errors about a type that exists nowhere in main's source; they vanished after a rebuild, with no source changeType of Change
Related Issues
None. Fallout from the work in the auth-diagnostics PR; independent of it and mergeable on its own.
Changes Made
Broken commands that were documented
AGENTS.mdtold everyone to runpnpm run lint:fix, which does not exist at the root. There is also no rooteslint.config.mjs, so thenpx eslint <changed-paths> --fix --quietform (introduced by fix(infra): credential diagnostics and anonymous fallback for hub resolution #374's ownAGENTS.mdrewrite, and repeated in the plan I was following) fails there too.pnpm -C packages -r <script>looks scoped topackages/but-rstill resolves the whole workspace, so it also runs the extension,lib/,website/andgithub-actions/: 4920 lines of output where the scoped form gives 181.New root scripts (all filtering on
@ai-primitives-hub/*, the same filter CI already uses)build:packages,test:packages,lint:packages,lint:extensionlint:fix— packages + extension, so the long-documented command now existstest:extension— delegates to the extension's existingtest:all(compile-tests → unit → integration). My first cut hand-rolledcompile-tests && test:unit, which both duplicatedtest:alland silently dropped the integration layertest:extension:unit— the fast path (~20s, no VS Code) for the inner loopverify:packages(~30s) andverify(~132s), chaining everything in dependency order behind a single exit codeEvery pre-existing script is byte-identical —
build,watch,lint,test,compile,compile-tests,test:unit,package:vsix. I checked which ones CI invokes (buildin the publish workflow,lintandtest:unitinquick-check.sh,compile/compile-tests/test:integrationin the secure CI) and only added alongside them.Notably not done: a
pretest:unithook, which would make the staleness trap structurally impossible. CI runscompile → post-compile-fixes → compile-tests → post-compile-fixes-essential.js → test:unit, so a pretest hook would re-run the compile after those fixes are applied.test:unitkeeps its exact definition;test:extensionis the safe path instead. Closing the hole properly means pointing CI at the underlying Mocha invocation — a CI change, so it is left as a decision rather than assumed.Lint noise
noWarnOnMultipleProjectson the sharedeslint-import-resolver-typescriptconfig. Every package legitimately has both asrcand atesttsconfig, so each lint run printed aMultiple projects foundwarning to stderr that reads like a fault.Guides
AGENTS.md: rewritten Commands section plus a new Verification section covering the three traps, the two different test-runner summary formats (VitestTests N passedvs MochaN passing, and that ANSI codes break naive matching unless you pass--no-color), and the fact that extension tests log expected errors from negative-path cases that are not failures.packages/AGENTS.md: points at the scoped scripts; notes that a cross-package type change is invisible until a build.apps/vscode-extension/AGENTS.md: a table of what each test layer sees, and why a passing unit run says nothing about command registration orpackage.jsoncontributions.apps/vscode-extension/test/AGENTS.md:compile-testsbeforetest:unitis mandatory, not conventional..gitignore.gitnexus/, a local code-intelligence cache — ~111 MB of single-line JSON. One shellgrep -rof mine read it and returned a six-figure-character line. It was only excluded via this clone's.git/info/exclude, so nothing protected anyone else.Testing
No product code changes, so the testing here is that the documented commands actually run and that the composite scripts fail when they should.
Test Coverage
Manual Testing Steps
build:packages,test:packages,lint:packages,lint:extension,lint:fix,test:extension:unit(23s),test:extension(39s),verify:packages,verify— all exit 0.pnpm run verify→ PASS in 132s, running 2196 unit + 7 integration tests, with lint leaving no files rewritten.verifyfails correctly: injected a failing Vitest test,verify:packagesexited non-zero.pnpm run <script>named in the four guides exists in the rightpackage.json(accounting for the one guide whose commands run fromapps/vscode-extension/).git diff package.jsonshows no-line for any of the eight.Tested On
macOS
Windows
Linux
VS Code Stable
VS Code Insiders
VS Code Stable via
@vscode/test-electron, as part ofverify.Screenshots
Not applicable.
Checklist
No automated tests: the deliverable is scripts and guidance. Each claim in the guides was verified by running the command, and each trap was reproduced deliberately before being written down — details under Manual Testing.
Documentation
The
AGENTS.mdguides are the change; no other docs needed updating.Additional Notes
verifyrunslint:fix, which rewrites source files. That follows the house rule ("always run linting with its:fixoption"), but if you would rather have a read-only gate I can splitverify(checking) fromverify:fix(mutating). Say which you prefer.verifyrequires a display for the integration layer (CI wraps it inxvfb-runon Linux).verify:packagesandtest:extension:unitare the headless-safe subsets.One loose end I could not explain: the
AGENTS.mdinjected into my session context already contained this exact guidance, including the "4920 lines" figure — yet no branch, remote ref or stash in the repository contains that text, andmainhas the old version. Something in the local setup appears to serve a newerAGENTS.mdthan any committed state. Worth checking, since an agent reading guidance that is not onmainwill diverge from what CI and other contributors see.Reviewer Guidelines
Please pay special attention to:
verifyshould includelint:fix(mutating) or be split.pretest:unit, accepting a double compile in CI, sopnpm run test:unitcannot silently pass on stale artifacts. I chose not to touch a CI-critical path unasked; the trade-off is yours..github/workflows/is cheap insurance.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0.