diff --git a/README.md b/README.md index 529e36cf..8d10b3d0 100644 --- a/README.md +++ b/README.md @@ -154,7 +154,7 @@ For CI and evals, `config.sample.toml` includes `[engines.mock]` so the enforcem ![Identical workers, each under its own light](docs/engines.png) -Ringer ships with three worker lanes: **Codex CLI** is the built-in default, and `config.sample.toml` carries verified engine blocks for **Grok Build CLI** (works as-is once you `grok login`) and **OpenCode + OpenRouter** (one edit: point `bin` at the sandbox wrapper in your clone). Anything else with a headless CLI is a config block away: +Ringer ships with four worker lanes: **Codex CLI** is the built-in default, and `config.sample.toml` carries verified engine blocks for **Grok Build CLI** (works as-is once you `grok login`), **OpenCode + OpenRouter**, and **Claude Code** (the last two need one edit: point `bin` at the sandbox wrapper in your clone). Anything else with a headless CLI is a config block away: ```toml [engines.mymodel] @@ -206,6 +206,14 @@ grok login Route with per-task `"engine": "grok"` and pick the model with `"model": "grok-build"` or `"model": "grok-composer-2.5-fast"` (the shipped default — the speed pick). Grok brings its own OS sandbox on macOS (profile `workspace`: read everywhere, writes confined to the task dir, temp, and `~/.grok`), and its JSON output exposes no token counts — plan-billed workers report cost as included in plan. +### The free lane: Claude Code, optionally against a local endpoint + +`config.sample.toml` also carries a `[engines.claude]` block that runs the `claude` CLI headlessly. Point `bin` at an absolute path to `engines/claude-sandboxed.sh` (same Seatbelt design as the OpenCode wrapper: network and reads open, writes confined to the task dir, a per-run scratch dir, and claude's own state/config locations) and it works with your normal Anthropic auth. + +It also runs against **any Anthropic-compatible `/v1/messages` endpoint** — Ollama, LM Studio, LiteLLM, a self-hosted gateway — which makes it the lane with no API bill at all and no prompt leaving hardware you control. Endpoint config is machine-specific, so it stays out of the repo: the wrapper sources `~/.config/ringer/claude-engine.env` when that file exists, where you export `ANTHROPIC_BASE_URL`, `ANTHROPIC_AUTH_TOKEN`, and an empty `ANTHROPIC_API_KEY`. With no such file, plain `claude` auth applies. + +Two things the sample block does deliberately, both from probe evidence in [`docs/MODEL-NOTES.md`](docs/MODEL-NOTES.md): it passes `--bare` and `--strict-mcp-config`, because headless claude otherwise inherits the operator's whole interactive environment (MCP rosters, plugin hooks, auto-memory, global `CLAUDE.md`) and a small local model answers that injected context instead of the spec; and it documents that `--bare` honors `ANTHROPIC_BASE_URL`/`ANTHROPIC_AUTH_TOKEN` even though its `--help` text describes auth in that mode as API-key-only. One local model on one box serves one request at a time, so give a local-endpoint engine 1-2 tasks per batch. + `args_template` is an argv array, not a shell string. Ringer replaces `{taskdir}`, `{spec}`, and `{model}` inside each argv element. `{access_args}`, `{sandbox_args}`, `{full_access_args}`, `{model_args}` (becomes `-m ` when the task or engine names one), and `{engine_args}` (the task's per-task `engine_args`) expand to multiple argv elements only when they appear as their own array item. Watch for variadic CLI flags. If an engine has a flag that consumes all following values, put `{spec}` before that flag. For Claude-style CLIs, prefer: @@ -357,6 +365,7 @@ Every community PR that lands in main is credited here — that's a project rule - [@davekopecek](https://github.com/davekopecek) (Dave Kopecek) — committed the design-reference fixture so the design-token guard runs on every machine (#30) - [@snapsynapse](https://github.com/snapsynapse) (Sam Rogers) — graceful shutdown on SIGINT/SIGTERM with worker-tree cleanup and finished state, plus the 14-test end-to-end CLI regression suite (#4) - [@mlava](https://github.com/mlava) (Mark Lavercombe) — named setup failures across every diagnostic surface (#37) and `run --baseline`, the no-workers check preflight (#38) +- [@perpetualsec](https://github.com/perpetualsec) (Alex Cox) — the `claude` engine: headless Claude Code workers, incl. free local inference via Anthropic-compatible endpoints Contributions are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md) for the philosophy and what gets a PR merged fast. The short version: small and scoped, rebased on current main, every claim backed by an executed test. Authorship is always preserved — where a maintainer pushes a mechanical fix to your branch, you remain the commit author. diff --git a/config.sample.toml b/config.sample.toml index 1527dead..96ba1896 100644 --- a/config.sample.toml +++ b/config.sample.toml @@ -170,6 +170,77 @@ model_report_regex = "(?m)^model:[ \\t]*([^ \\t\\r\\n]+)[ \\t]*\\r?$" # full_access_args = ["--no-sandbox"] # token_regex = '"tokens":\{"total":([0-9]+)' +# Claude Code (`claude`) as a headless worker. Two ways to use this engine: +# 1. Normal Anthropic auth — your usual `claude` login/API key. Leave the +# endpoint alone and set model_default to an Anthropic model slug. +# 2. Any Anthropic-compatible /v1/messages endpoint — Ollama, LM Studio, +# LiteLLM, a self-hosted gateway. This is the free-worker lane: real model +# output with no API bill (the mock engine's no-bill purpose, but with an +# actual model), and the prompt never leaves hardware you control. +# Endpoint config is machine-specific, so it lives OUTSIDE the repo: the wrapper +# sources ~/.config/ringer/claude-engine.env if that file exists, e.g. +# export ANTHROPIC_BASE_URL="http://your-ollama-host:11434" +# export ANTHROPIC_AUTH_TOKEN="local" # any non-empty value; local servers +# # generally ignore it +# export ANTHROPIC_API_KEY= # empty: no silent fallback to a real +# # key or to keychain/OAuth auth, +# # which is unreachable headlessly +# With no such file, plain `claude` auth applies. Set `model` per task (or +# model_default here) to whatever slug the endpoint serves — for Ollama, the +# local tag, e.g. "qwen3.6:latest". +# claude has no OS sandbox of its own (headless -p needs +# --dangerously-skip-permissions), so `bin` points at +# engines/claude-sandboxed.sh (macOS Seatbelt: network + reads open, writes +# confined to the task dir, a per-run scratch dir, and claude's own state/config +# locations: ~/.claude, ~/.claude.json, ~/.cache/claude). +# +# --bare and --strict-mcp-config are both load-bearing for small local models, +# and are here rather than in the wrapper so a fully-configured Claude Code +# worker stays possible. Probe evidence (2026-07-24, 27B qwen3.6 over Ollama — +# see docs/MODEL-NOTES.md): headless claude otherwise inherits the operator's +# whole interactive environment — MCP server rosters, plugin hooks, auto-memory, +# global ~/.claude/CLAUDE.md — and a small model answers that injected context +# instead of the task spec (an unrelated MCP tool call in one probe, a +# hallucinated blocker copied from prior-session context in the next). --bare +# skips hooks, plugin sync, auto-memory, and CLAUDE.md auto-discovery; +# --strict-mcp-config with no --mcp-config loads zero MCP servers. Together they +# also drop harness overhead from 16k-32k input tokens per turn to ~1.2k. +# Note: --bare honors ANTHROPIC_BASE_URL + ANTHROPIC_AUTH_TOKEN even though its +# --help text documents only ANTHROPIC_API_KEY/apiKeyHelper auth (verified +# 2026-07-24, claude CLI on macOS) — so mode 2 above works under --bare. +# Latency/concurrency caveat for local endpoints: one local model on one box +# serves one request at a time, so parallel Ringer lanes just queue behind it. +# Give such an engine 1-2 tasks per batch; wider only inflates wall time. +# Uncomment and set an absolute path for `bin` to enable. +# [engines.claude] +# bin = "/absolute/path/to/ringer/engines/claude-sandboxed.sh" +# model_default = "qwen3.6:latest" +# args_template = [ +# "{taskdir}", +# "{access_args}", +# "-p", +# "--output-format", +# "json", +# "--dangerously-skip-permissions", +# "--bare", +# "--strict-mcp-config", +# "--model", +# "{model}", +# "{engine_args}", +# "{spec}", +# ] +# sandbox_args = [] +# full_access_args = ["--no-sandbox"] +# Claude's JSON has no single combined total-tokens field the way Codex and +# OpenCode do; usage reports input_tokens and output_tokens separately. +# output_tokens is the meaningful per-attempt count — what the model actually +# generated. input_tokens is dominated by harness context and swings with that, +# not with task difficulty. +# token_regex = '"output_tokens":\s*([0-9]+)' +# modelUsage.canonicalModel self-reports the exact slug served, so scoreboard +# attribution stays exact even when the endpoint is a local server. +# model_report_regex = '"canonicalModel":"([^"]+)"' + [artifact] # Zero-LLM, self-contained HTML artifacts rendered directly from state — no model calls, no # network. Three files per config, all pure stdlib templates: diff --git a/docs/MODEL-NOTES.md b/docs/MODEL-NOTES.md index d7306dc2..ea2f0e70 100644 --- a/docs/MODEL-NOTES.md +++ b/docs/MODEL-NOTES.md @@ -312,3 +312,42 @@ checks and raw logs support — no vibes, no worker self-reports. ## opencode / z-ai glm-5.2 (via openrouter) - 2026-07-09 (aicred-invoice-downloads, 4 code-fix tasks + 1 follow-up, worktrees+npm ci checks): systematic attempt-1 NO-OP — all 4 parallel workers produced zero edits and no summary on first attempt, then completed cleanly on attempt 2 after retry-prompt injection (34k-69k tokens each). Follow-up single task passed attempt 1. Suspect first-invocation session warm-up in opencode-sandboxed under parallel spawn; budget for 2 attempts on parallel GLM batches. Output quality on Next.js/Stripe route+test work: solid, spec-faithful, one boss-caught design gap (used user-scoped supabase client where RLS demanded service role — spec didn't say explicitly; say it explicitly). + +## Small local models via the claude engine (Anthropic-compatible endpoints) + +- 2026-07-24 — probe (new `claude` engine, a 27B qwen3.6 served by Ollama over a + LAN, `claude -p --output-format json`): two failed probe cycles before the + cause was isolated, then a pass. The failures were not the model's reasoning — + they were harness context. Headless `claude` inherits the operator's entire + interactive environment, and a small model cannot tell that injected + background apart from the task spec: + - Cycle 1 (no `--strict-mcp-config`): the worker inherited the operator's full + MCP server roster, made an unrelated tool call, hit a schema error, and + abandoned the task — its final answer was an essay on an unrelated topic. + Neither required file was written. + - Cycle 2 (`--strict-mcp-config` only): MCP tool calls stopped, but + `--strict-mcp-config` does not stop plugin hooks or the always-loaded global + `~/.claude/CLAUDE.md`. The worker hallucinated a blocker that reads as + prior-session context bleeding in (an unrelated host being unreachable), + then ran past the 32k output-token ceiling without finishing. + - Cycle 3 (`--bare` + `--strict-mcp-config`): PASS on attempt 2 of the same + one-task manifest, with an executed check (run the generated script, compare + output and SHA-256) confirming the artifact — not a worker self-report. + Before rewiring anything, an isolated `claude -p --bare` "reply PONG" probe + against the same endpoint verified the mode: harness input dropped from + ~16k-32k tokens per turn to ~1.2k, and the reply was correct. +- Transferable lessons, both of which apply to any model behind this engine, not + just this one: + - Pass `--bare` **and** `--strict-mcp-config` when the worker is a small model. + `--bare` is the one that actually fixes it (it skips hooks, plugin sync, + auto-memory, and CLAUDE.md auto-discovery); `--strict-mcp-config` is + belt-and-suspenders, since worker tasks have no business calling the + operator's MCP servers regardless. The token drop is a side benefit — the + point is that the spec stops competing with injected context. + - `--bare` honors `ANTHROPIC_BASE_URL` + `ANTHROPIC_AUTH_TOKEN` even though its + `--help` text documents auth in that mode as strictly `ANTHROPIC_API_KEY` or + `apiKeyHelper` (verified 2026-07-24, claude CLI on macOS). The help text + reads like a blocker for alternative-endpoint setups and is not one. +- Latency/concurrency: one local model on one box serves one request at a time, + so parallel lanes queue behind it. Give a local-endpoint engine 1-2 tasks per + batch; wider inflates wall time with no real concurrency. diff --git a/engines/claude-sandboxed.sh b/engines/claude-sandboxed.sh new file mode 100755 index 00000000..2522f461 --- /dev/null +++ b/engines/claude-sandboxed.sh @@ -0,0 +1,140 @@ +#!/bin/bash +# Ringer engine wrapper: run the `claude` CLI (Claude Code) under a macOS +# Seatbelt sandbox as a headless Ringer worker. +# +# claude has no OS-level sandbox of its own (headless -p mode needs +# --dangerously-skip-permissions to run without interactive approval prompts), +# so this wrapper supplies the real containment: full network and reads, writes +# confined to the task dir, a per-run scratch/cache dir, and claude's own +# state/config locations (~/.claude, ~/.claude.json, and its small +# ~/.cache/claude staging dir). +# +# Usage (as a ringer engine bin): +# claude-sandboxed.sh [--no-sandbox] +# +# The first argument is the task directory (pass "{taskdir}" first in +# args_template). "--no-sandbox" as the second argument skips Seatbelt entirely +# — wire it as the engine's full_access_args so ringer's allow_full_access gate +# still applies. macOS only (sandbox-exec); on other platforms only +# --no-sandbox mode works. +# +# --------------------------------------------------------------------------- +# Inference endpoint: two modes, both supported, nothing machine-specific here +# --------------------------------------------------------------------------- +# +# 1. Default (no env file): plain `claude` auth applies — your normal Anthropic +# API key / login, exactly as an interactive `claude` run would resolve it. +# +# 2. Alternative Anthropic-compatible endpoint: if +# ~/.config/ringer/claude-engine.env exists it is sourced (before the sandbox +# is entered, so no profile rule is needed to read it) and its exports are +# inherited by claude. Anything that speaks the Anthropic /v1/messages API +# works — Ollama, LM Studio, LiteLLM, a self-hosted gateway. Typical file: +# +# # ~/.config/ringer/claude-engine.env +# export ANTHROPIC_BASE_URL="http://your-ollama-host:11434" +# export ANTHROPIC_AUTH_TOKEN="local" # any non-empty value; local +# # servers usually ignore it +# export ANTHROPIC_API_KEY= # empty: no silent fallback to a +# # real Anthropic key or to +# # keychain/OAuth auth, which is +# # unreachable non-interactively +# +# Keep that file out of version control — it is machine config, not repo +# code. Endpoint/model choice belongs there and in your config.toml's +# [engines.claude] block, never in this script. Set +# RINGER_CLAUDE_ENGINE_ENV to source a different path (handy for testing +# both modes without moving the real file). +# +# --------------------------------------------------------------------------- +# Why the engine's args_template should carry --bare and --strict-mcp-config +# --------------------------------------------------------------------------- +# +# Both are passed by the caller (see the [engines.claude] sample block in +# config.sample.toml), not by this wrapper, so a fully-configured Claude Code +# worker is still possible when that is what you want. They matter most for +# small local models: headless claude otherwise inherits the operator's whole +# interactive environment — MCP server rosters, plugin hooks, auto-memory, and +# the always-loaded global ~/.claude/CLAUDE.md — and a small model answers that +# injected context instead of the task spec. --bare skips hooks, plugin sync, +# auto-memory, and CLAUDE.md auto-discovery; --strict-mcp-config (with no +# --mcp-config supplied) loads zero MCP servers. See the entry in +# docs/MODEL-NOTES.md for the probe evidence. +set -euo pipefail + +TASKDIR="${1:?usage: claude-sandboxed.sh [--no-sandbox] }"; shift +SANDBOX=1 +if [ "${1:-}" = "--no-sandbox" ]; then SANDBOX=0; shift; fi + +# Resolve claude without tripping `set -e` (command -v returns nonzero when absent). +if ! CLAUDE_BIN="$(command -v claude)" || [ -z "$CLAUDE_BIN" ]; then + echo "claude-sandboxed.sh: claude not found on PATH" >&2 + exit 127 +fi + +# Optional machine-local endpoint config. Absent by default; when absent, plain +# `claude` auth applies. See the header for the file's shape. +CLAUDE_ENGINE_ENV="${RINGER_CLAUDE_ENGINE_ENV:-${XDG_CONFIG_HOME:-$HOME/.config}/ringer/claude-engine.env}" +if [ -f "$CLAUDE_ENGINE_ENV" ]; then + # shellcheck source=/dev/null + . "$CLAUDE_ENGINE_ENV" +fi + +if [ "$SANDBOX" = "0" ]; then + exec "$CLAUDE_BIN" "$@" < /dev/null +fi + +if [ ! -x /usr/bin/sandbox-exec ]; then + echo "claude-sandboxed.sh: /usr/bin/sandbox-exec not available (macOS only)." >&2 + echo "Use the engine's full-access mode (--no-sandbox) or add your own sandbox." >&2 + exit 1 +fi + +TASKDIR_REAL="$(cd "$TASKDIR" && pwd -P)" + +# Per-run scratch root — becomes both TMPDIR and XDG_CACHE_HOME for claude, so +# we never have to open all of /private/tmp or ~/.cache to the sandboxed agent. +# Resolve to the real path (/var/folders symlinks to /private/var/folders); +# Seatbelt subpath matching needs the canonical path or writes EPERM-crash. +SCRATCH="$(cd "$(mktemp -d -t ringer-claude-scratch)" && pwd -P)" +PROFILE="$(mktemp -t ringer-claude-prof)" +cleanup() { rm -rf "$SCRATCH" "$PROFILE"; } +trap cleanup EXIT + +# Paths are passed to the profile via sandbox-exec -D parameters, NOT string +# interpolation — a task dir containing quotes/parens/newlines can't inject rules. +cat > "$PROFILE" <<'SBEOF' +(version 1) +(allow default) +(deny file-write*) +(allow file-write* + (subpath (param "TASKDIR")) + (subpath (param "SCRATCH")) + (subpath (param "CLAUDE_HOME")) + (subpath (param "CLAUDE_CACHE")) + (literal (param "CLAUDE_JSON"))) +; /dev is needed for /dev/null, /dev/urandom, etc.; writes there can't create +; persistent files without root, so a few literals are allowed rather than via param. +(allow file-write-data + (literal "/dev/null") + (literal "/dev/dtracehelper") + (literal "/dev/tty")) +SBEOF + +export TMPDIR="$SCRATCH" +export XDG_CACHE_HOME="$SCRATCH/cache" +mkdir -p "$XDG_CACHE_HOME" + +# Run as a child (not exec) so the EXIT trap fires and cleans up the profile + +# scratch dir even on the success path; propagate the child's exit status. +set +e +/usr/bin/sandbox-exec \ + -D "TASKDIR=$TASKDIR_REAL" \ + -D "SCRATCH=$SCRATCH" \ + -D "CLAUDE_HOME=$HOME/.claude" \ + -D "CLAUDE_CACHE=$HOME/.cache/claude" \ + -D "CLAUDE_JSON=$HOME/.claude.json" \ + -f "$PROFILE" "$CLAUDE_BIN" "$@" < /dev/null +status=$? +set -e +exit "$status"