test(tools): make test:live actually test something - #19
Merged
Conversation
`IX_LIVE_TESTS` appeared in exactly three places -- the `test:live` script, a CI comment and a docstring -- and no test ever read it. There was no gate for the variable to open, so `bun run test:live` ran the identical 91 offline tests, and no tool had ever been exercised against a real `ix`. That left every tool covered only on its fallback path: the suite drives them at a directory where `ix` cannot succeed and asserts they return a string rather than throwing. A regression in argument shape, output parsing or exit-code handling would pass the whole suite. Add a `LiveIxCli` block behind `describe.skipIf(!IX_LIVE_TESTS)` covering ix-stats, ix-health and a deliberate ix-locate miss. The load-bearing assertion is the absence of each tool's unavailable marker, which is what separates a real success from the fallback the offline tests already cover. Note ix-health uses its own marker (`Status: UNAVAILABLE`, not the `ix unavailable` string the others use). Asserting the wrong one made that test pass with no `ix` on PATH at all -- verified each test fails when `ix` is absent, so none of them is vacuous. Also corrects the CI comment and docstring, which described an @LiVe suite that did not exist. Closes #18
#20 landed first and appended its own `NonZeroExitDiagnostics` block to the end of `tests/tools.test.ts`; this branch appends `LiveIxCli` to the same place. Git saw one conflict region covering both additions. They are independent -- `LiveIxCli` is opt-in behind `IX_LIVE_TESTS` and drives the tools against a real `ix`, while `NonZeroExitDiagnostics` runs a stubbed `ix` in a child process -- so both are kept, and both sets of imports survived the merge. bun test: 119 pass, 3 skip, 0 fail.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #18.
Problem
IX_LIVE_TESTSappeared in exactly three places — thetest:livescript, a CI comment, and a docstring — and no test ever read it. There was no gate for the variable to open:So no tool had ever run against a real
ix. Every test drives the tools at/tmp/ix-test-nonexistent-workspace-99999, which only proves the fallback path — that a tool returns a string instead of throwing. A regression in argument shape, output parsing, or exit-code handling would pass the entire suite.Change
A
LiveIxCliblock behinddescribe.skipIf(!IX_LIVE_TESTS), coveringix-stats,ix-health, and a deliberateix-locatemiss (a miss is a real answer — conflating it with an unavailable CLI is exactly the kind of bug only a live run catches).The load-bearing assertion is the absence of each tool's unavailable marker, since that is what separates a real success from the fallback already covered offline.
The part worth reviewing
ix-healthuses a different marker —Status: UNAVAILABLE, not theix unavailablestring every other tool emits. My first version asserted the wrong one, and that test passed with noixon PATH at all.So I checked each test both ways:
ixpresentixabsentNone of them is vacuous. Offline (
bun test) is 91 pass / 3 skip / 0 fail, unchanged.Verified against
ix0.10.5 and backend 1.0.27; live suite stable over three consecutive runs (~1.2s).Also corrects the CI comment and the test docstring, which both described an
@livesuite that did not exist.