Skip to content

ci: assert every hebb command in plugin.json / hooks.json / docs resolves to a real Click command #30

Description

@ch-liuzhide

Context

Goal axis: 好用 / usability — permanently close the "advertised command does not exist" regression loop that both audits kept rediscovering.

The original failure (audit core-system-audit-2026-06-07.md, finding C0-2) was that .claude-plugin/plugin.json and .codex/hooks.json invoked a command that did not exist: the group is registered as @click.group("claude-code") (so hebb cc ... resolves to "No such command"), and there was never a write subcommand (UserPromptSubmit should call prompt). Meanwhile the programmatic installer wrote the correct strings, so two install paths silently drifted apart — breaking recall + memory for anyone who installed via the plugin marketplace or that shipped hooks.json. The newuser audit (newuser-experience-audit-2026-06-08.md, finding U3) found the same hebb claude-code write ghost command duplicated across the docs (cli.md, claude-code.md, README).

PR #24 fixed the strings, but a string fix does not prevent the next drift. This issue is about the durable guard.

Current state (verified against the tree)

A guard for the manifest + Dockerfile surface already landed in PR #24 (commit 9244265) — so this issue is narrowed to the remaining gap (docs / examples / READMEs), not the whole thing.

What already exists and passes (pytest tests/unit/test_audit_distribution.py → 3 passed):

  • tests/unit/test_audit_distribution.py:97 test_plugin_hook_commands_resolve — extracts every hook command from .claude-plugin/plugin.json and resolves each against the live Click tree.
  • tests/unit/test_audit_distribution.py:107 test_codex_hook_commands_resolve — same for .codex/hooks.json.
  • tests/unit/test_audit_distribution.py:117 test_dockerfile_cmd_references_real_serve_command — same for the docker/Dockerfile CMD chain.
  • tests/unit/test_audit_distribution.py:26 _resolve_command(...) — the shared Click-tree walker (from hebb.cli.main import main); reused below.
  • This file is already in CI: .github/workflows/ci.yml:51-55 runs pytest tests/unit ... on every OS/Python matrix cell.

Grounding of the shipped strings (all correct today, confirming PR #24 landed):

  • .claude-plugin/plugin.json:20,32,44hebb claude-code recall / prompt / stop.
  • .codex/hooks.json:9,18,27hebb claude-code recall / prompt / stop.
  • src/hebb/integrations/claude_code/cli.py:8@click.group("claude-code"); subcommands at lines install (20), uninstall (34), recall (42), prompt (50), stop (58). Live introspection confirms claude-code subs = install, prompt, recall, stop, uninstall.

The remaining gap — no test resolves hebb ... strings embedded in human-facing docs, which is exactly where the write ghost command lived (U3):

  • No test under tests/ references repo_pages, examples, or README (grep is empty).
  • repo_pages/**/*.md contains ~109 hebb <subcommand> strings (e.g. repo_pages/guide/claude-code.md:25,42,43,49), with an EN tree and a repo_pages/zh/ mirror.
  • examples/03_mcp_quickstart.md and examples/README.md contain hebb ... command strings (e.g. examples/03_mcp_quickstart.md:22,25,33).
  • README.md and README_ZH.md both carry hebb ... invocations.

So a future doc typo (hebb claude-code wrtie, a renamed/removed subcommand, etc.) would ship green today.

Proposed approach

Extend the existing tests/unit/test_audit_distribution.py (reuse its _resolve_command walker and from hebb.cli.main import main introspection — do not duplicate the tree logic) with a docs/examples/README coverage test:

  1. Walk README.md, README_ZH.md, examples/**/*.md, and repo_pages/**/*.md (skip repo_pages/node_modules/).
  2. Extract every hebb ... invocation from both fenced code blocks (```bash/```sh/plain) and inline `hebb ...` spans.
  3. Normalize each invocation to a token path (strip the hebb executable; stop the sub-command path at the first -/-- option or shell metachar, matching the existing _hebb_invocations helper).
  4. Assert each resolves to a registered command + subcommand via _resolve_command. Allow a small, explicit allowlist for intentionally illustrative/placeholder strings (e.g. hebb ... literally, or hebb <command> help syntax) so the test stays low-noise.
  5. The test runs in the existing CI test job automatically (it lives under tests/unit).

Keep the manifest/Dockerfile tests as-is; this only adds the docs/examples/README dimension.

Acceptance criteria

  • A test introspects the live Click command tree from the hebb entrypoint (reuses the existing _resolve_command / main import, no duplicate tree logic).
  • It extracts and resolves hebb ... invocations from README.md, README_ZH.md, examples/**/*.md, and repo_pages/**/*.md (EN + repo_pages/zh/ mirror), covering both fenced blocks and inline code spans.
  • Introducing a fabricated bad command (e.g. hebb claude-code wrtie or hebb foo bar) in any covered doc fails the test; the current tree passes.
  • The manifest coverage (.claude-plugin/plugin.json, .codex/hooks.json) and Dockerfile coverage remain green (no regression to the existing three tests).
  • The new test runs in CI on every PR (verified via .github/workflows/ci.yml test job picking up tests/unit).
  • Any intentional placeholder strings are handled via a narrow, documented allowlist rather than broad regex loosening.

Scope / out of scope

  • In scope: a CI-enforced assertion that every hebb ... string advertised in shipped manifests and human-facing docs/examples/READMEs resolves to a real Click command+subcommand.
  • Out of scope: validating option flags/argument values (e.g. that --scope user is a legal choice) — only the command-path is resolved, matching the existing _resolve_command semantics. Out of scope: localized prose translation correctness, and non-hebb shell commands in fences.
  • Note: the manifest + Dockerfile half already shipped in PR fix(core): 0.1.7 — audit remediation + eval harness + custom HTTP embedding #24; this issue is the doc/example/README extension only.

This is the highest-leverage usability fix in the backlog because it permanently closes the doc/command-drift regression loop that the core-system-audit-2026-06-07 (C0-2) and newuser-experience-audit-2026-06-08 (U3) audits independently rediscovered.

References

  • reports/audit/core-system-audit-2026-06-07.md (finding C0-2 — hebb cc write / wrong group name)
  • reports/audit/newuser-experience-audit-2026-06-08.md (finding U3 — hebb claude-code write doc drift)
  • reports/design/capability-gap-roadmap-2026-06-11.md
  • tests/unit/test_audit_distribution.py (existing manifest/Dockerfile guard to extend)

Filed from the capability-gap roadmap (reports/design/capability-gap-roadmap-2026-06-11.md).

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: usabilityInstall, first-run, docs UX / 好用effort: simpleQuick win: ~0.5-2 days, low risk, high ROIenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions