██
 ▄▄ ▄▄▄  ▄▄▄▄   ▄▄▄▄  ▄▄  ▄▄  ▄▄▄▄  ██
 ███▀▀▀ ██▄▄██ ██▄▄██ ██  ██  ▄▄▄██ ██
 ██     ██   ▄ ██   ▄  ████  ██  ██ ██
 ██      ▀▀▀▀   ▀▀▀▀    ▀▀    ▀▀▀▀▀ ▀▀

 Evaluation runner for SDKs, CLIs, MCP servers, and docs.

 # Get started using the `/setup` skill:
 npx skills add invertase/reeval --skill setup

 # Or install manually:
 npm i -D reeval && npx reeval init

Important

Work in Progress: reeval is under active development. Expect breaking changes between minor version releases.

reeval

reeval is an open-source evaluation runner for developer tools. It gives a real coding agent a task in a fresh sandbox, then checks the result with assertions on files, commands, and the transcript, and with an LLM judge where an assertion can't decide. Run the same task with and without a skill, doc, or MCP server, and reeval reports whether the change helped.

Evals are TypeScript files in evals/, next to the tool they test. Each run writes a JSON record, and recorded runs commit to Git, so reeval compare can diff a pull request's run against the last recorded one and fail the job when an eval regresses.

Features

Cursor ⋅ Claude Code ⋅ Codex ⋅ Antigravity ⋅ OpenCode ⋅ Goose ⋅ pi ⋅ Grok Build

• Real agents: each harness runs the agent itself, with its own tools, skills, and rules, as it works on a developer's machine (Choose an agent).
• Keys stay on the host: each attempt starts in a fresh sandbox that holds only a placeholder key. reeval's egress gateway swaps in the real key on requests to that provider and nowhere else (Pass secrets).
• Compare a change: one eval runs a baseline and its variants with the same prompt and checks, then prints whether each variant helped, made it worse, or can't tell yet (Measure lift against a baseline).
• Pinned environments: an environment starts from a workspace, such as a git repository at a fixed ref, then adds files, setup steps, rules, skills, and MCP servers, so every attempt starts from the same state (Define an environment).
• History in Git: recorded runs sit beside the code. reeval query reads them with SQL, and reeval studio shows each attempt's transcript, diff, and checks (Record runs in Git).

Install

npm i -D reeval

reeval runs on Node.js 22.13 or later. Real runs also need a sandbox runtime (Docker or Podman) and an API keys for the agents. npx reeval doctor checks both and names the fix for anything missing.

Quick start

Scaffold a config and one eval, then check that reeval loads them:

npx reeval init --yes     # reeval.config.ts on Cursor, and evals/greet.eval.ts
npx reeval run --dry-run  # smoke check: plans the run, no keys or sandbox

plan  run … · 1 eval · 1 cell · 1 attempt · ~3m 00s serial
eval                                    harnesses  variants  runs  attempts  notes
evals/greet.eval.ts › creates greet.js  1          —         1     1
…

Then build the agent's sandbox image, add its key, and run it for real:

npx reeval image build
echo 'CURSOR_API_KEY=…' >> .env.local
npx reeval run

Without --yes, init asks which agent to configure. The starter eval only proves the setup works; the next section is what reeval is for.

Measure whether a skill helps

The Vercel AI SDK deprecates generateObject in favor of Output.object, and agents trained on older code still reach for the old call. This eval measures whether a skill shipped with the SDK fixes that. It runs the same task in two arms, four attempts each: plain has no skill, and with-skill adds the one in skills/ai-sdk.

// evals/ai-sdk.eval.ts
import { defineEnvironment, evaluate, expect } from 'reeval';

const plain = defineEnvironment('plain', {});

const withSkill = plain.extendEnvironment('with-skill', ({ skill }) => ({
  skills: [skill.local('../skills/ai-sdk')],
}));

evaluate(
  'triages a GitHub issue with the AI SDK',
  { runs: 4, compare: { baseline: plain, variants: [withSkill] } },
  async (c) => {
    await c.send(`
      Add triage.ts: triageIssue(title, body) uses the ai package to
      return a typed { labels, priority, duplicateOf }.
    `);

    await expect(c, 'writes triage.ts').toHaveFile('triage.ts');
    await expect(c, 'uses Output.object').toHaveFile('triage.ts', /Output\.object/);
  },
);

$ npx reeval run evals/ai-sdk.eval.ts
…
pass             triages a GitHub issue with the AI SDK
  arm                 pass
  plain               0 of 4
  with-skill          4 of 4  helped
  check               plain   with-skill
  uses Output.object  0 of 4  4 of 4
    expected triage.ts to match /Output\.object/
  +1 more passed
…
1 pass

Without the skill, no attempt used Output.object; with it, all four did. helped means the 95% interval of the difference sits above zero (-v prints it), and made it worse that it sits below. When the interval spans zero, or an arm has too few attempts, reeval prints can't tell yet. +1 more passed folds the checks that every arm passed.

Run in CI and inspect runs

Command	Use it for
reeval run --profile pr	A run sized for a pull request. In the config init writes, that is one run per cell, no variants, and at most 20 attempts.
reeval compare --fail-on-regression	Diff a run against its nearest recorded ancestor, and exit 1 when an eval regressed.
reeval show --md	One run's results as Markdown, for a job summary or a PR comment.
reeval query	Read-only SQL over every run, or built-in views such as lift, effort, and failing_attempts.
reeval studio	Runs, transcripts, diffs, and checks in the browser, or a static site with reeval studio build.

Environments load extensions from where teams already keep them:

• Skills: from a local folder, a git repository at a pinned ref, an npm package, the skills.sh registry, or written inline.
• MCP servers: stdio, Streamable HTTP, or SSE servers, with their own environment variables and headers.
• Rules: the project's instructions, written as AGENTS.md, or CLAUDE.md on Claude Code, before the first turn.

Examples

The recipes in docs/evaluations/recipes are complete evals for common jobs:

• Measure a skill: with and without a skill, gated on a minimum lift.
• Compare harnesses: one suite on several agents at once, with pass rate and effort per agent.
• Measure MCP servers: check the agent calls the server's tools, and whether answers improve.
• Check a recommendation: ask which library to use, and let a judge decide whether the agent picks the right one.
• Fail a pull request: run on every pull request and fail the check on a regression.

The examples/ directory has 80 runnable projects, one idea each, from with-checks (deterministic checks) and with-skill-lift (does a skill help?) to sandboxes, egress, judges, captures and real-world tasks. Most run on a fake agent with no keys or Docker; examples/README.md lists them all.

Documentation

The manual in docs/ covers harnesses, sandboxes, environments, secrets and egress, checks and judges, profiles, history, and every CLI command and flag.

For coding agents

To write or update evals, load skills/reeval/SKILL.md. For every command, flag, and exit code, run npx reeval --help.

To preview a run without starting sandboxes or spending tokens, run npx reeval run --dry-run --json. It prints the planned evals, cells, and attempts as one JSON document and exits 0.

To check a machine's setup, run npx reeval doctor --json. It checks Node.js, the sandbox and its images, and the agents' keys, never prints a key, and changes nothing.

To read past runs without calling a model, run npx reeval query failing_attempts --json, or npx reeval analyze --context for a run's digest. npx reeval mcp serves the run history over stdio as read-only MCP tools.

License

Apache-2.0. Copyright 2026 Invertase Limited. The npm package ships third-party licenses and attributions in THIRD_PARTY_NOTICES.md. Questions about licensing: oss@invertase.io.