[umbrella] Runtime independence v1 — expose runner as an independently-usable component

[CiroGamboa]

Context

Runner-independence work — expose the runtime component of tolokaforge as an independently-usable library so external harnesses (custom pipelines, third-party frameworks, agent labs) can pip install tolokaforge, bring their own docker image, plug in their own grader/tools, and drive the runner as an agent from outside the CLI. Design captured in docs/architecture/ROADMAP.md (Planned row: "Runner as an independently-usable component") and docs/architecture/RUNTIME_BACKENDS.md § "Extending to new substrates" + § "Follow-up work". Related public discussion in #156 ([design] Library-shaped composition).

Locked design decisions

• Not a multi-package split — "Same package, same wheel."
• Delivery via entry-point registry over the three Protocols (RuntimeBackend, TrialGrader, Conductor) — plus a slim runner Docker image + a tolokaforge agent CLI subprocess contract.
• Top-level tolokaforge.run_trial(...) entry point for library-shaped consumption.

Sub-issue decomposition — deferred

Sub-issues will be filed via /writing-development-tickets when this milestone is actively started — the design is already stable, but converting it into 5-6 PR-sized sub-issues is best done at kickoff so each maps cleanly to a /executing-development-tickets invocation. Candidate scope from the design docs:

• Design ADR (may reuse [design] Library-shaped composition — expose agentic loop, grader, user simulator as independently callable components #156).
• Expose RuntimeBackend / TrialGrader / Conductor as entry-point extension groups.
• tolokaforge agent CLI subprocess-contract mode.
• Slim runner Docker image (no full tolokaforge.core deps).
• Top-level tolokaforge.run_trial(...) entry point.
• Integration verification — external harness imports tolokaforge and drives a trial end-to-end.

Not blocked by

The current Multi-container runtime v1 milestone. Both tracks can run in parallel; a natural ordering is to close Multi-container runtime v1 first (small, immediate) before tackling this (larger, more architectural).

Constraints

• No AI attribution in commits or PR bodies.
• No internal repo/adapter names or plan-doc IDs in public content.
• One PR per sub-issue when filed.
• No release bumps as part of this milestone — the runner-independence surface is additive to what's already installed.

[CiroGamboa]

Sub-issue decomposition — filed (kickoff, milestone #13)

Per this umbrella's "Sub-issue decomposition — deferred" section, the six PR-sized sub-issues have been filed and assigned to this milestone. Ordering by dependency (keystone first):

1. docs(adr): runtime independence — Protocol registries + tolokaforge agent subprocess contract #535 — docs(adr): runtime independence — Protocol registries + tolokaforge agent subprocess contract
2. feat(runtime): expose RuntimeBackend / TrialGrader / Conductor as entry-point extension groups #536 — feat(runtime): expose RuntimeBackend / TrialGrader / Conductor as entry-point extension groups
3. feat(runtime): top-level tolokaforge.run_trial(...) library entry point #537 — feat(runtime): top-level tolokaforge.run_trial(...) library entry point
4. feat(cli): tolokaforge agent subprocess-contract mode for external harnesses #538 — feat(cli): tolokaforge agent subprocess-contract mode for external harnesses
5. feat(docker): slim runner Docker image with runner-only wheel subset #539 — feat(docker): slim runner Docker image with runner-only wheel subset
6. test(integration): external-harness end-to-end proof of runtime independence #540 — test(integration): external-harness end-to-end proof of runtime independence

Milestone driven via /implement-milestone on integration branch feat/runtime-independence-v1; per-issue PRs squash-merge into that branch; a single consolidation PR to main at the end.

Design context reused from #156 ([design] Library-shaped composition) as noted in the umbrella.

[CiroGamboa]

#535 → PR #543 → a1848bf — ADR-0019 (Runtime independence) landed as Proposed on feat/runtime-independence-v1. Docs-only; four surfaces locked (Protocol registries #536, run_trial #537, agent subprocess #538, versioning+slim-image+lifecycle #539/#540). Follow-ups filed: #541 (RUNTIME_BACKENDS.md method-count drift, P3), #542 (ADR-index/ROADMAP link-integrity guard, P3). Nit: reserved-auto prose can be tightened when #536 implements Surface 1.

[CiroGamboa]

#536 → PR #546 → 49c5bba — entry-point Protocol registries landed on feat/runtime-independence-v1. Three importlib.metadata groups (tolokaforge.runtime_backends / .trial_graders / .conductors), fail-loud registry loader, six built-ins registered, orchestrator + CLI + config validate rewired through the registry; endpoint helpers pushed down into shared_stack_runtime.py with boundary invariant (no impl module imports orchestrator) grep-enforced. OrchestratorConfig.runtime widened Literal → str; docker legacy alias preserved. Follow-ups filed: #544 (adapters registry silent last-wins), #545 (runtime banner reads deprecated field). Nit deferred: three config-validator tests could move canonical → unit (reviewer flagged non-gating). CI hygiene-review lane failed as pre-existing infrastructure noise (also red on PR #531); lint + test-smoke green.

[CiroGamboa]

#537 → PR #549 → 1a7173c — top-level tolokaforge.run_trial(...) library entry landed on feat/runtime-independence-v1. Frozen keyword-only ADR-0019 § Surface 2 signature; three ADR-named error types (UnknownImplementationError / ValidationError / ProvisionError) all raise loudly; runtime="auto" mirrors task-driven selection; output_dir=None writes nothing to disk; run-level connect()/close() pairing guaranteed in a finally. Canonical composition-equivalence lock (InMemory + InMemoryConductor, no Docker/LLM, every PR) + integration real-agent-loop parity lock (shared + in_process, push/nightly/gate). Follow-ups: #547 (public tolokaforge.load_task(...) helper, P2), #548 (shared provision bracket, P3). Corrective fix during review: run-level runtime_backend.close() was leaking a gRPC channel (and Docker stack for manifest tasks) — fixed via try/finally. CI hygiene-review lane failed as pre-existing infra noise; lint + test-smoke green.

[CiroGamboa]

#538 → PR #552 → fb29d7d — tolokaforge agent subprocess-contract mode landed on feat/runtime-independence-v1. JSON-Lines wire format at "v":1 (ADR § 4a); reads start or cancel message on stdin, calls run_trial(...), writes exactly one result or error on stdout with typed error_type mapping (traceback → stderr, non-zero exit). SIGTERM (POSIX) + premature stdin EOF + synchronous cancel → error_type:"cancelled". Wire-authoritative selection (no CLI runtime/grader/conductor flags). fd-level stdout redirect isolates the wire from StructuredLogger (which binds stdout on the shared surface). tolokaforge run byte-for-byte unchanged. Follow-ups: #550 (mid-trial cancel + Windows, P3), #551 (event progress subtypes, P3). Corrective fold during review: v1 envelope hardening (unknown keys → ProtocolError), annotation honesty, dead-class cleanup, example-import consistency. CI hygiene-review lane failed as pre-existing infra noise; lint + test-smoke green.

[CiroGamboa]

#539 → PR #554 → 89de7e1 — slim runner image (4 stages + 1 corrective). Measured 390.2 MB, 40.8% reduction from 659 MB baseline — beats ADR §4b ≥40% target compat-safely. [runner] extra as SSOT replaces dead [docker] extra + hand-list; multi-stage Dockerfile with --no-compile (79 MB bytecode strip); INSTALL_DOCKER_CLI=false opt-in with auto-detection hook mirroring Playwright (terminal-bench task auto-gets docker CLI back — :local compat preserved). Boot check green; import-boundary canonical test locks the surface; 5 E2E slim-image tests including litellm-in-container (the user's 'absolutely sure' lock) + real tolokaforge run tool_use smoke succeeded end-to-end. ADR-0019 §4b amended with measured figure + litellm-floor rationale. Docs updated: docs/RUNNER.md new image-contents section; docs/RUNTIME_BACKENDS.md; docs/PYTHON_PACKAGE.md dead [docker] ref fixed; docs/FUTURE_DEVELOPMENT.md Issue 1/Stage 9 moved to Completed Work. Discovery surprise: original issue premise was wrong — runner does import litellm via core.grading.judge (105 MB floor). Follow-up filed: #553 (test_docker_grading_jsonpath.py broken by register_trial param drift during #536/#537, P2). Corrective fold during review: enable_docker_cli unit tests + docs/FUTURE_DEVELOPMENT.md current-state rewrite + size-test dedup. CI hygiene-review pre-existing infra failure; lint+test-smoke green.

[CiroGamboa]

#540 → PR #556 → 5900e02 — MILESTONE #13 CAPSTONE. External-harness end-to-end proof + ADR-0019 flipped Proposed → Accepted. All 6 milestone issues merged into feat/runtime-independence-v1; ready for consolidation PR to main.

Milestone summary:

• docs(adr): runtime independence — Protocol registries + tolokaforge agent subprocess contract #535 → a1848bf — ADR-0019 (contract source, 4 surfaces)
• feat(runtime): expose RuntimeBackend / TrialGrader / Conductor as entry-point extension groups #536 → 49c5bba — entry-point Protocol registries + fail-loud loader
• feat(runtime): top-level tolokaforge.run_trial(...) library entry point #537 → 1a7173c — tolokaforge.run_trial(...) library entry
• feat(cli): tolokaforge agent subprocess-contract mode for external harnesses #538 → fb29d7d — tolokaforge agent subprocess-contract mode (JSON-Lines "v":1)
• feat(docker): slim runner Docker image with runner-only wheel subset #539 → 89de7e1 — slim runner image (390.2 MB / 40.8% reduction)
• test(integration): external-harness end-to-end proof of runtime independence #540 → 5900e02 — external-harness E2E capstone + ADR §Accepted

Capstone locks: downstream plug-in package resolves all 3 seams from installed metadata; drives trials via both run_trial() (byte-equal parity vs Orchestrator baseline, no mask) and tolokaforge agent (byte-stable against checked-in golden). Slim-image + agent subprocess capstone drives #538+#539+#536+#537 together. Load-bearing matrix explicit: removing any one of #536-#539 fails at least one test.

All follow-ups filed during the milestone: #541 (RUNTIME_BACKENDS method-count drift, P3), #542 (ADR-index link-integrity guard, P3), #544 (adapters registry silent last-wins, P3), #545 (runtime banner reads deprecated field, P3), #547 (public tolokaforge.load_task helper, P2), #548 (shared provision bracket extraction, P3), #550 (agent mid-trial cancel + Windows, P3), #551 (agent event progress subtypes, P3), #553 (test_docker_grading_jsonpath broken by register_trial param drift, P2).

Ready for the consolidation PR to main.

[CiroGamboa]

Milestone 13 complete. Consolidation PR #557 merged to main (merge commit c2f8fbe on the integration branch, integrated into main via the PR merge).

All six sub-issues delivered:

• docs(adr): runtime independence — Protocol registries + tolokaforge agent subprocess contract #535 — ADR keystone (now landed as ADR-0022; renumbered from 0019 on integration to avoid collision with upstream's front-end plugin namespace ADR).
• feat(runtime): expose RuntimeBackend / TrialGrader / Conductor as entry-point extension groups #536 — Entry-point Protocol registries (tolokaforge.runtime_backends / .trial_graders / .conductors).
• feat(runtime): top-level tolokaforge.run_trial(...) library entry point #537 — tolokaforge.runner.run_trial(...) library entry (namespaced under runner during integration).
• feat(cli): tolokaforge agent subprocess-contract mode for external harnesses #538 — tolokaforge run-trial subprocess CLI (renamed from agent → run-one → run-trial during integration to fix collision with models["agent"] AI-role sense; final CLI location under tolokaforge.dx.cli.run_trial_command per upstream ADR-0019 refactor).
• feat(docker): slim runner Docker image with runner-only wheel subset #539 — Slim runner Docker image (390 MB, -40.8% from 659 MB baseline).
• test(integration): external-harness end-to-end proof of runtime independence #540 — External-harness E2E capstone + ADR flip Proposed → Accepted.

Successor milestone M14 (runner as a distributable service) is planned and unblocked on the Docker Hub side — see ~/.claude/plans/lively-giggling-narwhal.md. Three review-first ADR tickets (#573 secrets courier, #578 wheel split, #579 container command surface) gate M14-B.