Run wasm tools in hub test instead of skipping them

[jorgeMFS]

The problem

hub test looks only for runtime/native/{bin}, prints [SKIP] when it isn't there, and returns a pass.

25 of the 32 recipes have no native build. So for 78% of the catalogue, a green test run says only that the build produced a file of the expected name. Anything that links cleanly and then traps at run time goes unnoticed — that is how four newick-utils tools reached review, and why kallisto took three blind CI rounds.

What this does

Runs the wasm artifact when there is no native binary. A small Node harness (hub/tests/run_wasm.js) loads the module, writes the generated inputs into its filesystem, calls main, and returns stdout, stderr and whatever files the tool left behind. The Python side writes those into the temporary directory, so the existing output discovery and type checking apply unchanged — every recipe is exercised the same way, whichever runtime it ships.

The harness returns the whole working directory rather than a named list, because the test never tells a tool where to write: it looks afterwards for a file named after the output. The same discovery has to work for both runtimes.

One build flag is required

The recipes are compiled with ENVIRONMENT=web,worker, and such a module refuses to load anywhere else:

FAILS: not compiled for this environment (did you build to HTML and try to run it not on the web?)

That is the actual reason wasm tools could only ever be skipped. Adding node:

build	flags	Node
A	ENVIRONMENT=web,worker (current)	fails to load
B	ENVIRONMENT=web,worker,node	loads and runs

It costs ~3 KB of JS shim (168,381 → 171,410 bytes on a test tool) and changes nothing for the browser targets.

A module that cannot run now fails

Previously an aborting module still produced VERDICT: PASS. A load failure or abort is now a test failure. A tool's own non-zero exit is still only reported, matching how the native path treats it, since some tools exit non-zero by design.

Verification

Against artifacts built with the hub's own EM_FLAGS, using emsdk 4.0.18 (the version the recipes pin):

case	result
working wasm-only tool	passes, and genuinely runs (Testing tool democopy (wasm))
truncated .wasm	fails with the load error
tool emitting the wrong output type	fails — Detected: UNKNOWN, Expected: ['FASTA']
native tool	passes, still receiving absolute paths — unchanged
neither artifact present	still skips, not a failure

The third case is the one that matters: it proves the type checking has teeth through the new path rather than passing vacuously.

Run against real recipes

Built with the hub's own pipeline (validate → build → test) on emsdk 4.0.18, for the five wasm-only recipes that use the emscripten strategy — every one of them previously [SKIP]ped:

recipe	before	after
edlib	SKIP	runs, passes
abpoa	SKIP	runs, passes
cgranges	SKIP	runs, passes
fermi-lite	SKIP	runs, passes — but [WARNING] Empty output
tn93	SKIP	runs, passes — but [WARNING] Output file not found

So five real tools now execute where nothing did before, and none of them regress CI.

Two things this exposes, which are worth knowing before merging:

1. The pass criterion is still weak. fermi-lite and tn93 run and produce nothing usable, and still pass, because empty or missing output is only a [WARNING]. That is pre-existing behaviour and I have not changed it here — the cause is most likely that the generated example input is not meaningful for those tools rather than the tools being broken. Making it fail would need better per-type fixtures first, otherwise it would just turn recipes red without telling anyone anything useful.

2. Where a native build exists, the wasm artifact is still never tested. Native takes precedence, which preserves existing behaviour, but the wasm build is what the SPA actually runs. That affects ksw2, lastz, prodigal and samblaster. Arguably both should run; I have kept the change minimal.

Not covered here: the 22 biowasm/auto recipes, which carry 167 of the 176 operations and need Docker to build. Their behaviour under this change is unknown until CI runs it, and that is the real blast radius — a 9-operation sample cannot speak for 167.

Note this PR also needs #33 (lazy Docker client) to be buildable without a daemon, though the two are independent as changes.

Review found three things that would have made this harmful

Worth reading before the rest, because they change what the PR is:

1. hub test could not fail. test_cmd printed the failing tools and returned, so the command exited 0 and CI passed regardless. Everything here was log text. It now exits non-zero — verified: exit 1 on a failing tool, 0 on a skip.

2. Nineteen recipes would have flipped from skipped to failing, blaming the recipe. Only the emscripten builder gained node; the other nineteen are built by the biowasm container, whose flags live in its own bin/shared.sh and were out of reach of the original change. Those modules abort with not compiled for this environment — which says nothing about whether the tool works.

Two changes: the harness reports that refusal distinctly, and the test skips it with a plain reason rather than failing; and the biowasm image is patched at build time to add node, so those tools become testable once the image is rebuilt.

3. -1 was both a sentinel and a real exit status. return -1 is an ordinary way for a tool to reject its arguments, and it was indistinguishable from a trap — so the tool's output was discarded and it was reported broken. Loading and completion are separate fields now.

Also fixed: non-UTF-8 stderr raised UnicodeEncodeError out of print(), which escaped and left every remaining recipe untested; and each tool is now tested inside its own handler so one cannot end the run.

case	result
browser-only module (biowasm-style)	skips, does not fail
genuinely broken .wasm	fails with a real diagnostic
tool returning -1	keeps its output, not mistaken for a trap
non-UTF-8 stderr	run completes
hub test with a failing tool	exit 1
real edlib, rebuilt	still passes

Honest limits

• The biowasm Dockerfile change is unverified — it needs Docker, which I do not have. Until that image is rebuilt, those nineteen recipes skip with an explicit reason rather than silently, which is still an improvement on the previous behaviour but is not the same as testing them.
• Separately, that Dockerfile clones biowasm unpinned (git clone ... . with no ref), so what it builds depends on when the image was made. Not addressed here.
• The pass criterion remains weak: [WARNING] Empty output and a missing output file do not fail, and no output flag is ever passed to a tool. So for a large share of operations the check still cannot go red. Strengthening it needs better per-type fixtures and belongs in its own change.

Review notes

• Independent of the R work (Add a CI job that cross-compiles R packages to WebAssembly #27–Build recipes that declare the R strategy #29) — different files, and it branches from master.
• The native execution path is deliberately untouched: it still builds absolute paths and still ignores the tool's return code. Both are arguable, but changing them risks the seven recipes that currently rely on it, and that belongs in its own change.
• Existing published artifacts were built without the node flag. This matters only if hub test is ever run against a registry pull rather than a fresh hub build; in CI the build immediately precedes the test, so the artifacts carry the flag.
• node becomes a requirement for hub test. It is already present on the runners, and the failure mode is an explicit message rather than a crash.

---

Merge order

sbom-implementation is the priority branch and rewrites parts of the builders. This PR conflicts with it on hub/builders/emscripten.py. Rebase after it lands. The resolution is one line — -s ENVIRONMENT=web,worker becomes -s ENVIRONMENT=web,worker,node; EM_FLAGS is otherwise unchanged there.

Independent of the R work (#27, #28, #29) and of #25/#26.

Interaction with #36

Checked against sbom-implementation as it stands on 2026-08-10, since #36 is now open.

• It does not touch hub/tests/, so nothing here overlaps it.
• Its EM_FLAGS still reads -s ENVIRONMENT=web,worker, so the one-line resolution described above still applies.
• It does not change test_cmd, so the SystemExit(1) added here is still needed and does not conflict — both branches edit hub/hub.py in different places and merge cleanly.

Worth knowing while both are in flight: hub test appears in validate-recipes.yml but not in publish-recipes.yml, which runs validate → build → sbom → sbom-check → publish → sign-attest. So on the publishing path the tools are never executed, and on the PR path the command currently exits 0 whatever happens. Until this PR lands, a bundle can be signed and attested without any tool in it having been run successfully. That is an observation about how the two fit together, not a claim about #36.