Add a Rook CLI skill for testing AI agents by what they actually changed #761

Request

Add skills/rook/ so an OpenHands agent can test an AI agent the user owns with Rook CLI, TestMu AI's command-line tool for Agent Assurance.

Disclosure: I work at TestMu AI (formerly LambdaTest), which builds Rook CLI. Related: #758 and #760, the TestMu AI MCP catalog entry.

The gap

The registry can review code and pull requests, and qa-changes validates a change by running it. Nothing tests an AI agent itself: invoking it on generated scenarios, including adversarial ones such as prompt injection and tool misuse, and grading what it actually changed rather than what it reports.

What Rook CLI does

• Reads a repository or requirements docs to work out what the agent does, then writes functional, non-functional and adversarial scenarios.
• Invokes the real agent and grades each criterion against observed evidence: files changed, artifacts produced, and tool calls checked against the agent's declared tools.
• Reports three verdicts, Pass, Fail and Unable to Verify, and keeps the third out of the pass rate.
• Runs headless in CI. Exit 0 means the run finished, so a gate reads the verdicts from rook report --json.
• Apache-2.0, https://github.com/LambdaTest/rook, latest release v0.1.7. Pre-alpha. Needs a TestMu AI account, and runs spend Agent Assurance credits.

Proposed change

• skills/rook/SKILL.md and skills/rook/references/, adapted from the upstream skill at the v0.1.7 tag (https://github.com/LambdaTest/rook/tree/v0.1.7/.agents/skills/rook), with no links to main.
• skills/rook/README.md.
• An entry in marketplaces/openhands-extensions.json, plus the files scripts/sync_extensions.py generates.

The skill will follow the points raised on #477:

• Pinned install: npm install -g @testmuai/rook@0.1.7 (Node 22 or newer), never --force or @latest.
• The user signs in and approves the target agent and any credit spend. The agent never approves those itself. Headless runs read LT_USERNAME and LT_ACCESS_KEY from OpenHands secrets.
• It is offered as an option for testing agents, not as a replacement for any built-in tool.
• Plain hyphens throughout.

How it will be tested

In an OpenHands sandbox with the released package: install @testmuai/rook@0.1.7, sign in, run Rook CLI against the public refund-desk sample in the repo, and attach the run summary to the PR, with no credentials or account data.

OpenHands AI triage

The following comments and acceptance criteria were added by the OpenHands AI agent.

Triage

This adds a new progressive-disclosure skill, skills/rook, that lets an agent test an AI agent the user owns with Rook CLI (TestMu AI's Agent Assurance tool). The repository has no existing implementation of agent testing; qa-changes runs changed code but does not test an agent itself. The request is a coherent, self-contained extension submission and needs no unresolved product decision. The upstream source is a public v0.1.7 tag and the @testmuai/rook@0.1.7 package exists on npm. No design question remains open, so this is ready for development.

Repository conventions apply from AGENTS.md and README.md: a skill directory with SKILL.md and README.md (tests/test_skills_have_readme.py), one marketplace entry using a valid skill category (automations, environment, code-hosting, agent-authoring, code-quality, integrations, writing, design, other), the .plugin/plugin.json manifest plus .claude-plugin/.codex-plugin vendor symlinks, and every derived artifact regenerated by the existing sync/build commands (python scripts/sync_extensions.py, npm run build:skills). The marketplace entry must be listed in marketplaces/openhands-extensions.json so test_marketplace_includes_all_skills passes, and should omit defaultEnabled because this is vendor-specific, aligning with the defaultEnabled guidance.

Bounded scope: create skills/rook/ from the upstream v0.1.7 skill, add its README.md, add one marketplace entry, and regenerate the derived files. Explicit non-goals: this skill is offered as an option for testing agents and does not replace qa-changes or any built-in tool; it makes no change to plugins/qa-changes; it does not modify SDK or Agent Server behavior; and it does not document an unshipped Rook feature. Repository punctuation style requires plain hyphens, not em dashes.

The readiness sections the repository guidance expects (### Actual Behavior, ### Steps to Reproduce, ### Desired Behavior) are bug-report headings; this is a feature request, so an ### Expected behavior section is recorded here instead.

Expected behavior

Installing OpenHands and enabling the rook skill lets an agent drive Rook CLI to discover an owned agent, generate scenarios, run them against the real agent, grade the observed evidence, and report Pass, Fail and Unable to Verify separately. The skill is opt-in, uses a pinned @testmuai/rook@0.1.7 install and references pinned to the v0.1.7 tag, requires no links to main, and leaves existing skills and tools unchanged.

Acceptance Criteria

• skills/rook/SKILL.md exists with valid YAML frontmatter (name, description) that parses with PyYAML and loads through the SDK Skill.load().
• skills/rook/references/ is present and every reference link in SKILL.md resolves to a file in the skill directory.
• skills/rook/README.md exists and is non-empty.
• The upstream content is adapted from the v0.1.7 tag; no URL or reference in skills/rook/ points to main.
• SKILL.md documents the pinned install npm install -g @testmuai/rook@0.1.7 and Node 22 or newer, and never documents --force or @latest.
• SKILL.md states that the user signs in and authorizes the target agent and any credit spend, that the agent never authorizes those itself, and that headless runs read LT_USERNAME and LT_ACCESS_KEY from OpenHands secrets.
• SKILL.md presents the skill as an option for testing agents and does not present it as a replacement for qa-changes or any built-in tool.
• marketplaces/openhands-extensions.json contains one entry for ./skills/rook with a category from the valid list, a description, and keywords, and without defaultEnabled.
• skills/rook/.plugin/plugin.json exists with name, description, author (an object with a name) and version, and .claude-plugin and .codex-plugin resolve to it.
• python scripts/sync_extensions.py --check exits 0 after regenerating artifacts, and npm run build:skills regenerates skills/index.js with no further diff.
• uv sync --group test && uv run pytest -q passes, including test_marketplace_includes_all_skills, test_skills_have_readme, test_plugin_manifest, and the SDK skill-loading tests.
• A live sandbox run installs @testmuai/rook@0.1.7, signs in, and runs Rook CLI against the public refund-desk sample; the run summary is attached to the PR with no credentials or account data, and no links to main appear in any added file.
• No secret, API key, access key, customer data or private endpoint is committed in skills/rook/ or the marketplace entry.