|
| 1 | +# Claude Code Skill guidance regression suite |
| 2 | + |
| 3 | +This project evaluates the packaged `powercontext-project-context` Skill with |
| 4 | +[skill-up v0.12.0](https://github.com/alibaba/skill-up/releases/tag/v0.12.0), pinned upstream source |
| 5 | +`dd2ff3250074505204b969405bef2cdce4bae428`. It measures **Claude Code + MCP instruction/tool-selection behavior** |
| 6 | +with deterministic `rule_based` assertions and controlled replies. Model behavior can still vary between runs. |
| 7 | + |
| 8 | +It complements the [integration guidance qualification record](../../docs/en/development/integration-guidance-evaluation.md). |
| 9 | +It neither replaces those multi-surface observations nor contributes to the SWE-bench Pro or LoCoMo scores. |
| 10 | +Offline validation and synthetic tests do not establish real model coverage; retain authenticated runs' full evidence. |
| 11 | + |
| 12 | +## Prerequisites and installation |
| 13 | + |
| 14 | +Use Python 3.11+, Git, Node.js, Bash, skill-up **0.12.0**, and an installed/authenticated Claude Code CLI with access to |
| 15 | +`claude-sonnet-4-6`. `python3`, `node`, and `claude` must be on PATH. The shell examples below target Linux/macOS |
| 16 | +or WSL; native Windows also supports model runs with the Bash/PATH setup below. The `none` |
| 17 | +runtime invokes host commands and is not an OS sandbox. Use a disposable evaluation account/workspace: the runner |
| 18 | +disables hooks and bypasses Claude Code permission prompts. A paid Claude model run executes both arms (12 user turns). |
| 19 | + |
| 20 | +From this directory on Linux x86-64 (Python setup also works on macOS/WSL): |
| 21 | + |
| 22 | +```bash |
| 23 | +python3 -m venv .venv |
| 24 | +. .venv/bin/activate |
| 25 | +python3 -m pip install -r requirements-dev.txt |
| 26 | +mkdir -p .venv/skill-up |
| 27 | +curl -fsSL https://github.com/alibaba/skill-up/releases/download/v0.12.0/skill-up_0.12.0_linux_amd64.tar.gz \ |
| 28 | + -o .venv/skill-up/skill-up.tar.gz |
| 29 | +printf '%s %s\n' ec2631e45702d460b69cd700e1847f5562ef2e971f09c23be2efcec4e0f583c6 \ |
| 30 | + .venv/skill-up/skill-up.tar.gz | sha256sum --check - |
| 31 | +tar -xzf .venv/skill-up/skill-up.tar.gz -C .venv/skill-up |
| 32 | +export PATH="$PWD/.venv/skill-up:$PATH" |
| 33 | +skill-up --version |
| 34 | +``` |
| 35 | + |
| 36 | +Other platforms must use their matching v0.12.0 release asset and verify its published checksum. Authenticate Claude |
| 37 | +Code using its supported login/API-key setup, then check `claude --version` and `claude auth status`. Do not store tokens |
| 38 | +in this project or pass them as command-line arguments. |
| 39 | + |
| 40 | +On native Windows, create the virtual environment with `python -m venv .venv` and install the same requirements using |
| 41 | +`.\.venv\Scripts\python.exe -m pip install -r requirements-dev.txt`. The Scope helper needs `python3` to resolve to that |
| 42 | +environment too; copying its interpreter within the same `Scripts` directory supplies that name. In PowerShell, |
| 43 | +adjust the installation paths, retaining Node.js on PATH: |
| 44 | + |
| 45 | +```powershell |
| 46 | +$evalVenv = (Resolve-Path .venv).Path |
| 47 | +Copy-Item -LiteralPath "$evalVenv\Scripts\python.exe" -Destination "$evalVenv\Scripts\python3.exe" |
| 48 | +$evalGit = 'C:\tools\PortableGit' |
| 49 | +$env:SKILL_UP_BASH = "$evalGit\usr\bin\bash.exe" |
| 50 | +$env:CLAUDE_CODE_GIT_BASH_PATH = "$evalGit\bin\bash.exe" |
| 51 | +$env:PATH = "$evalVenv\Scripts;C:\tools\skill-up;C:\tools\claude;$evalGit\bin;$env:PATH" |
| 52 | +``` |
| 53 | + |
| 54 | +The two Bash paths serve skill-up and Claude Code respectively. |
| 55 | + |
| 56 | +## Validate and run |
| 57 | + |
| 58 | +No PowerContext Server, database, MCP token, or readiness request is needed for this **mocked** suite. skill-up owns the |
| 59 | +Node stdio MCP process lifecycle, starting a fresh mock in each case/arm workspace. Responses are stateless, so a save |
| 60 | +cannot contaminate another case or run. The fixture Scope `skill-up-fixture-scope` is fictional. Keep `parallelism: 1`. |
| 61 | + |
| 62 | +```bash |
| 63 | +python3 sync_skill.py --check |
| 64 | +python3 validate_suite.py |
| 65 | +python3 -m unittest discover -s tests -v |
| 66 | +skill-up validate evals/eval.yaml |
| 67 | +python3 run.py --dry-run |
| 68 | +python3 run.py --output-dir results/first-live-run |
| 69 | +``` |
| 70 | + |
| 71 | +`run.py` is the acceptance entry point. It verifies the pin and runner version, requires a new output directory, sets |
| 72 | +`CLAUDE_PLUGIN_ROOT` to `harness/plugin-root`, and invokes |
| 73 | +`skill-up run evals/eval.yaml --baseline --iteration 1 --output-dir <directory>`. It records tool versions, input hashes |
| 74 | +and exit status, copies exact inputs into `inputs/`, then checks archived sessions even when the runner fails. |
| 75 | +To recheck a retained run without changing its original evidence: |
| 76 | + |
| 77 | +```bash |
| 78 | +python3 report.py --iteration results/first-live-run/iteration-1 \ |
| 79 | + --skill-lock results/first-live-run/skill-lock.json --output-dir results/rechecked --engine-exit-code 0 |
| 80 | +``` |
| 81 | + |
| 82 | +Replace `0` with the original runner exit code. Use the pin tested at execution time, never today's pin for an older run. |
| 83 | +Retain the whole output directory: provenance, pin, input snapshot, native reports, raw JSONL and qualification reports. |
| 84 | + |
| 85 | +## Cases and controls |
| 86 | + |
| 87 | +| Case | Required behavior | |
| 88 | +| --- | --- | |
| 89 | +| `ordinary-coding` | Answer the self-contained coding task and call no PowerContext tool. | |
| 90 | +| `explicit-save` | Call `mcp__powercontext__remember_memory` in the resolved fixture Scope; avoid unrelated tools. | |
| 91 | +| `empty-search` | Call search; report the controlled empty result; do not inventory or write. The following coding turn uses current context only. | |
| 92 | +| `inspect-candidates` | List and inspect the candidate; never approve, reject, revise or publish it. | |
| 93 | +| `failed-save` | Attempt the explicitly requested save, report `FIXTURE_WRITE_DENIED`, and do not claim persistence. | |
| 94 | + |
| 95 | +The ordinary-coding negative and explicit-save positive controls are inseparable. Full mock catalogs are retained in |
| 96 | +the failed-save override because v0.12.0 replaces the entire server fixture; dropping a forbidden tool could otherwise |
| 97 | +make its negative assertion pass trivially. Case files are listed explicitly: v0.12.0 does not expand `cases.files` |
| 98 | +globs. Even single-turn cases use `input.turns`; turn-scoped rules do not work with its single-shot `input.prompt` path. |
| 99 | +All relative config paths start at this project root, not at `evals/`. |
| 100 | + |
| 101 | +Both arms receive the same `evals/fixtures/repos/powercontext-tool-contract/CLAUDE.md`: required/optional top-level |
| 102 | +argument names, types and defaults derived from OpenAPI. This fills the generic mock schemas' parameter information gap |
| 103 | +without providing routing rules, fixture values or answers. Refresh it with `python3 validate_suite.py --write-tool-contract` |
| 104 | +after reviewing an API change. Validation rejects reference drift or overridden case context. |
| 105 | + |
| 106 | +`with_skill` means the pinned Skill was installed and available; `without_skill` omits that installation. These labels |
| 107 | +do not prove the agent read the Skill's full body. Inspect recorded Skill invocations or file reads before making a |
| 108 | +claim about body consumption; ordinary coding does not require a Skill invocation. |
| 109 | + |
| 110 | +## Pinning a Skill revision |
| 111 | + |
| 112 | +`vendor/powercontext-project-context` contains the four unchanged packaged Skill files. `skill-lock.json` identifies |
| 113 | +their source commit, individual SHA-256 hashes and an aggregate hash of sorted `<sha256> <relative-path>\n` lines. |
| 114 | +Vendoring reads immutable Git blobs, and `.gitattributes` keeps the vendor bytes in LF form across platforms. |
| 115 | + |
| 116 | +To evaluate a new packaged Skill, first commit that change, then explicitly update and review the pin: |
| 117 | + |
| 118 | +```bash |
| 119 | +python3 sync_skill.py --revision HEAD |
| 120 | +python3 sync_skill.py --check |
| 121 | +``` |
| 122 | + |
| 123 | +`python3 sync_skill.py` replays the existing locked revision, which must be available in local Git history. `--check` |
| 124 | +needs no historical fetch: it validates vendor hashes and compares the current packaged Skill, normalizing checkout |
| 125 | +CRLF to Git LF. CI fails if the packaged Skill drifts without updating the pin. A changed pin requires a new model run. |
| 126 | + |
| 127 | +## Interpreting evidence and limits |
| 128 | + |
| 129 | +- **C1 — Authentication.** The enabled fixture uses `mode: mocked`; v0.12.0 ignores mock headers entirely. Authentication |
| 130 | + is not tested. `evals/fixtures/mcp/powercontext-http.example.yaml` is an unused real-HTTP reference: only `config_ref` |
| 131 | + can supply its `headers`. `POWERCONTEXT_CLAUDE_AUTHORIZATION` is the complete `Bearer <token>` value, whereas Python |
| 132 | + SDK configuration may take a bare token. Never put `headers` directly under `mcp.servers[]`: they are silently ignored. |
| 133 | +- **C2 — Exact names.** Rule matching is exact. Assertions expect `mcp__powercontext__<tool>`, with `powercontext` from |
| 134 | + the MCP server key. The supplement reads actual `tool_use` records from the final cumulative Claude JSONL, retaining |
| 135 | + IDs, arguments, line references and hashes; `result.json` supplies grades, not calls. Review the first real inventory. |
| 136 | + Uncalled catalog names are not observed names. Wrong namespaces or missing required calls fail qualification. |
| 137 | +- **C3 — Disabled hooks.** skill-up sets `disableAllHooks: true`, and `/v1/context/prepare` is not an MCP tool. This suite |
| 138 | + cannot observe bounded recall, automatic Capture or Flush, nor does it measure memory quality. |
| 139 | +- **C4 — Permission model.** The runner uses `bypassPermissions`. Scores measure instruction/tool selection, not real |
| 140 | + host confirmation dialogs, approval enforcement or permission UX. |
| 141 | +- **C5 — State.** Stateless mocks isolate PowerContext responses, not the host profile. Claude inherits user settings, |
| 142 | + native memory and global Skills/MCP; use a dedicated evaluation profile. Do not use `CLAUDE_CONFIG_DIR` to isolate it: |
| 143 | + v0.12.0 looks for transcripts under `HOME/.claude/projects` regardless. Real mode would require isolated Scopes/reset. |
| 144 | +- **C6 — Lifecycle.** `environment.type: none` starts no PowerContext Server. The current runner starts its mock itself. |
| 145 | + The unused HTTP reference is not a ready-to-run real suite: it requires a separately started Server, an authenticated |
| 146 | + readiness/tool-catalog check, dedicated Scopes and explicit cleanup before any real case is admitted. |
| 147 | +- **C7 — Plugin boundary.** The packaged Skill references `${CLAUDE_PLUGIN_ROOT}/scripts/workspace_scope.py`, outside its |
| 148 | + Skill directory. The wrapper deliberately provides `harness/plugin-root/scripts/workspace_scope.py`, a fixture that |
| 149 | + returns only the fictional Scope and rejects bind/clear operations. The Skill text is unchanged. This evaluates the |
| 150 | + guidance after controlled Scope resolution; it does not qualify the actual plugin helper or standalone packaging. |
| 151 | +- **C8 — Host boundary.** Only Claude Code + MCP is covered. Results do not qualify Codex, Hermes, WorkBuddy, OpenCode, |
| 152 | + Pi, OpenClaw, or the portable Agent Plugin. |
| 153 | + |
| 154 | +The built-in mock publishes generic descriptions and permissive argument schemas. The shared workspace reference |
| 155 | +supplies source-derived parameter signatures equally to both arms, but does not make the mock validate them. Only |
| 156 | +explicitly asserted arguments are graded; this is not a real API contract test. Failed writes are controlled |
| 157 | +application-error JSON inside a successful MCP text envelope, not protocol errors or network timeouts. |
| 158 | +Output patterns cover known false-success phrases, not all possible claims; review final answers too. |
| 159 | + |
| 160 | +Every logical turn must finish with an assistant `end_turn` response after all recorded tool calls have matching results. |
| 161 | +The supplement checks every relevant call's Scope so a valid call cannot hide an additional wrong-Scope call. |
| 162 | + |
| 163 | +Native artifacts live in `iteration-1/`: `result.json`, `report.xml` (JUnit), HTML and `benchmark.json`, plus per-case |
| 164 | +raw sessions under `<case>/<with_skill|without_skill>/outputs/agent/run/`. `--baseline` produces both arms in |
| 165 | +one invocation. Native benchmark pass rate averages assertion pass rates; the qualification supplement reports whole-case |
| 166 | +pass rates with its additional transcript checks and the loaded-minus-unloaded delta. A baseline behavioral |
| 167 | +failure is comparison data, while a missing/error baseline is incomplete evidence. Report all five cases and both arms, |
| 168 | +even when the Skill has no improvement or performs worse. Do not aggregate these scores with task-outcome benchmarks. |
| 169 | + |
| 170 | +The `Skill guidance validation` workflow runs the credential-free checks on relevant pull requests. The authenticated |
| 171 | +model command is a separate gate for a maintainer-provided Claude environment; a green offline job is not a model pass. |
0 commit comments