Skip to content

feat(engineering): add a verification stage and a second provider to hivemind - #1003

Open
Hanishchow wants to merge 4 commits into
alirezarezvani:devfrom
Hanishchow:hivemind-verify-layer
Open

feat(engineering): add a verification stage and a second provider to hivemind#1003
Hanishchow wants to merge 4 commits into
alirezarezvani:devfrom
Hanishchow:hivemind-verify-layer

Conversation

@Hanishchow

Copy link
Copy Markdown
Contributor

Summary

The verification stage

Every worker claim had to be re-checked by hand, which cost more than the delegation saved. scripts/oc-verify.mjs sends the claims to a verifier that re-derives them from the same files and tries to refute them. It returns {confirmed, refuted, unsupported, unverified, needs_review[]} — you read needs_review only; confirmed claims come back as a count rather than prose.

Four rules keep it from becoming a rubber stamp, and they're documented in SKILL.md:

  • The verifier runs on a different model from the worker. Same model means the same blind spots.
  • It sees the files, never the worker's reasoning.
  • It defaults to REFUTED when uncertain, because a false CONFIRMED gets acted on unchecked.
  • A verifier returning fewer verdicts than claims marks the remainder unverified, not passed.

The second provider

opencode's free tier is not dependable — models return provider 404s and empty bodies. scripts/or-worker.mjs speaks the identical one-JSON-line contract over OpenRouter, so oc-verify, oc-status and oc-aggregate work against it unchanged. Key comes from OPENROUTER_API_KEY or ~/.claude/.openrouter_key, is never logged, and never appears in output.

SKILL.md now includes a measured table of which models actually work: six free-model attempts failed for five different reasons, while the same free model answered a plain-text question correctly at zero cost. The rule that falls out is capability, not availability — structured output needs a paid model, prose does not.

The --json fix

Workers bury their JSON in fenced blocks after a paragraph of prose. oc-worker.mjs --json now extracts the value and gives one corrective retry, instead of every caller writing the same regex. Unit-tested against fenced, prose-prefixed, bare-array, bare-object and fence-without-language shapes.

Checklist

  • Targets dev
  • Frontmatter unchanged: name + description only
  • 239 lines, under the 500 limit
  • Anti-patterns and Cross-references sections preserved from the merged version
  • assets/commands/ layout untouched — additive only, nothing removed
  • All three scripts pass node --check

Scripts remain .mjs rather than stdlib Python, as in #979. Happy to adjust if that's now a problem.

…hivemind

Two weaknesses in the merged version, both found by running it.

Every worker claim had to be re-checked by hand, which cost more than the
delegation saved. oc-verify.mjs sends the claims to a verifier that re-derives
them from the same files and tries to refute them, deliberately on a different
model so its blind spots do not match the worker's. It returns counts plus a
needs_review list, so only refuted and unsupported claims need a human.

And opencode's free tier is not dependable: models 404 at the provider and
return empty bodies. or-worker.mjs speaks the identical one-JSON-line contract
over OpenRouter, so oc-verify, oc-status and oc-aggregate work unchanged.

oc-worker.mjs also gains --json, because workers bury their JSON in fenced
blocks after a paragraph of prose; it now extracts the value and gives one
corrective retry instead of every caller writing the same regex.
…i#979

Overwriting oc-worker.mjs with my copy dropped the // auditor:ignore-line note
added upstream after the merge, which would have reintroduced a CRITICAL
CMD-INJECT finding in the security gate. Restored verbatim.

oc-verify.mjs imports child_process for the same reason - it invokes
oc-worker.mjs to run the verifier - so it carries its own justification rather
than borrowing that one.

Verified with the repo's own skill_security_auditor.py --strict: PASS, 0 critical.
Gate G10 failed because the new agent file had no `name` in its frontmatter.
The agents already in this repo were given namespaced names after alirezarezvani#979 merged
(hive-scout, hive-coder, hive-tester), so this one is hive-verifier.

Verified with scripts/check_frontmatter.py: 0 errors, same single warning the
existing agent files produce.
Gate G3 derives the agent count from the tree and compares it against the
headline numbers in CLAUDE.md and the README badge. Adding one agent file makes
those read 118 where the tree now has 119.

CONTRIBUTING asks contributors not to touch index files, but G3 blocks any PR
that adds an agent without this, so the two hunks are here to make CI pass -
revert them and bump it yourself if you would rather.
@Hanishchow

Copy link
Copy Markdown
Contributor Author

CI status and one thing I need your call on.

Fixed after opening:

  • Security audit → SUCCESS. Overwriting oc-worker.mjs with my copy dropped the // auditor:ignore-line note added upstream after feat(engineering): add hivemind orchestration skill #979, which reintroduced a CRITICAL CMD-INJECT. Restored verbatim. oc-verify.mjs imports child_process for the same reason (it invokes oc-worker.mjs) so it carries its own justification rather than borrowing yours. Verified locally with skill_security_auditor.py --strict: PASS, 0 critical.
  • Frontmatter gate G10 → passing. The new agent file had no name. The existing agents were namespaced after feat(engineering): add hivemind orchestration skill #979 merged, so it's hive-verifier now. check_frontmatter.py: 0 errors, same single warning your other agent files produce.

Still red: G3 (counter derivation), and I don't think I should fix it.

Adding one agent file takes the derived agents count 118 → 119. I bumped the two places a contributor can reasonably touch:

  • CLAUDE.md "Current Scope" line
  • the README Agents-118-blue badge

But derive_counters.py also validates the same claim in:

  • .claude-plugin/marketplace.json
  • .codex-plugin/plugin.json

CONTRIBUTING says not to modify marketplace.json or .codex/, and those look generated by your sync workflows — so editing them would be both against the rules and pointless once sync runs. As it stands, any external PR that adds an agent can't pass G3 without breaking CONTRIBUTING.

Three ways I'm happy to go:

  1. You regenerate the derived counters after merge, and G3 is treated as expected-red here.
  2. You'd rather I bump all four anyway — say so and I'll push it.
  3. Drop the agent file entirely. oc-verify.mjs works with any agent id via --model; verifier.md is a convenience persona. Removing it keeps the counters untouched and the verification layer intact. This is my recommendation if you want the simplest merge.

VirusTotal is failing the same way it did on #979 — fork PRs don't get the API secret. Tessl I have no visibility into.

Happy to take whichever path is least work for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant