Skip to content

0016-runtime-backend-comparison.md

Latest commit

 

History

History
151 lines (104 loc) · 15.9 KB

0016-runtime-backend-comparison.md

File metadata and controls

151 lines (104 loc) · 15.9 KB
Raw

0016. Runtime backend comparison: shared vs per_trial

  • Status: Accepted
  • Date: 2026-07-03
  • Deciders: @CiroGamboa
  • Supersedes: —
  • Superseded by: —

Context and Problem Statement

Two RuntimeBackend implementations ship: SharedStackRuntimeBackend and PerTrialRuntimeBackend. Both satisfy the same Protocol (ADR-0007) and are selected by operators via orchestrator.runtime or the --runtime CLI flag. RUNTIME_BACKENDS.md documents the mechanics — isolation, failure modes, port allocation, timing — but is silent on questions operators ask when picking one:

  • Does the grade a task produces depend on which backend I run?
  • Does result aggregation differ?
  • What resource cost does per_trial add over shared, and when is that cost justified?
  • Which failure modes are mode-specific?
  • Where do I look in logs, metrics, and artifacts when something breaks — does that differ by mode?

This ADR fills those gaps. It documents the shipped state (both backends validated end-to-end) rather than proposing new work. It cites the empirical A/B run conducted on coding_public_example_01 under both modes to confirm grading equivalence.

Scope note — this ADR covers one of two independent axes

Runtime backends live on a 2×2 matrix, not a line:

  • Axis 1 — Stack lifecycle (this ADR): shared vs per_trial — is the substrate materialised once per run or once per trial?
  • Axis 2 — Stack composition (ADR-0018): built-in stack (engine's core_stack/full_stack defaults) vs task-declared stack (a task's environment_manifest compose file). Is the substrate composed of engine services or task-declared services?

The lifecycle axis and the composition axis are orthogonal — a run occupies one cell of the 2×2, and each cell has distinct trade-offs. ADR-0018 documents the composition axis end-to-end (with sequence diagrams for each case) and the 2×2 as a whole. This ADR focuses on grading equivalence, resource profile, and decision rubric along the lifecycle axis; those claims hold regardless of composition.

Decision Drivers

  • Operator clarity. Cost, throughput, and isolation are the real trade-off surface — capture them in one place so operators don't have to reverse-engineer from source.
  • Reviewer evidence trail. Future PR reviewers should be able to point at a durable statement that grades and aggregates are mode-agnostic, without re-deriving the code path each time.
  • Future substrate compatibility. When Kubernetes / Modal / microVM backends land, this comparison generalises to "shared-materialisation vs per-trial-materialisation" — the docker specifics belong in RUNTIME_BACKENDS.md, the isolation-axis semantics belong here.

Considered Options

  1. Extend RUNTIME_BACKENDS.md in-place with the missing sections — no separate ADR.
  2. Write ADR-0016 as this comparison + rubric doc, and add a "See also: ADR-0016" link from RUNTIME_BACKENDS.md.
  3. Write ADR-0016 and rewrite RUNTIME_BACKENDS.md to be pointer-only.

Decision

We will adopt Option 2. RUNTIME_BACKENDS.md keeps its mechanics-focused content; ADR-0016 owns the durable comparison across the axes below and the decision rubric. The two docs cross-link.

Grading and aggregation equivalence

The grade a task produces is independent of the runtime backend, by construction of the shipped code paths:

  • RunnerRPCTrialGrader.grade() (tolokaforge/core/trial_grader.py) calls runtime_backend.grade_trial() — a method every RuntimeBackend implementation satisfies with identical shape. SharedStackRuntimeBackend.grade_trial dispatches to the shared runner's gRPC endpoint; PerTrialRuntimeBackend.grade_trial dispatches to the per-trial runner client looked up by trial_id. Both routes hit the same runner-side GradeTrial service (tolokaforge/runner/service.py), which executes the same three-phase grading algorithm (hash-of-final-state / jsonpath assertions / optional LLM judge) with no branch on which backend called it. The runner is trial-centric, not infrastructure-centric.
  • Conductor phase _capture_final_state (tolokaforge/core/conductor.py) also routes through runtime_backend.get_state(trial_id) — same-shape dispatch, same runner-side handler. The final-state snapshot the grader reads is captured through the mode-specific runner instance but shaped identically.
  • Result aggregation (Orchestrator._generate_reports) operates on the in-memory Trajectory objects each trial produces. It does not consult which backend produced the trajectory; it treats every trajectory as opaque data.

Empirical confirmation. coding_public_example_01 was run repeats=3 in each mode with identical model configuration (openrouter → sonnet-4-6, temperature=0.0):

Mode Trial 0 Trial 1 Trial 2 Avg grade Total cost Avg latency
shared 0.783 0.783 0.783 0.783 $0.225 44 s
per_trial 0.783 0.783 0.783 0.783 $0.246 54 s

Grade convergence, not trajectory identity. All six trials scored exactly 0.783, but the underlying trajectories varied within LLM non-determinism (per_trial trial 0 took 8 turns / 8 tool calls; the other 5 trials took 5–6 turns / 7 tool calls). Grade equivalence is therefore the stronger property: the grade is robust to small trajectory perturbations, and the substrate contributes zero systematic bias to the outcome.

Caveat. Grading equivalence is deterministic modulo LLM stochasticity. Runs at non-zero temperature or with LLM-judge rubrics will still produce the same expected grade distribution across modes, but individual trials will diverge on the LLM axis, not on the substrate axis.

Resource profile

Order-of-magnitude framing, not exhaustive benchmarks — enough for capacity planning. Concrete numbers come from the A/B run above and the docker-events cross-check.

Axis SharedStackRuntimeBackend PerTrialRuntimeBackend
Long-lived containers Engine services (runner + db-service) up for the run Engine images built (:local alias applied), engine containers not started
Per-trial containers None (trials share the engine) One compose project per trial (runner + task-declared services), created on provision, destroyed on teardown
Docker networks One shared engine network (runner-net) for the whole run runner-net created (so :local aliases work) + one docker network per active trial
Port allocation Fixed set of host ports for the shared engine Testcontainers picks a fresh set of host ports per trial
Startup latency Engine build + start (~seconds on cached build, minutes on first run) Engine build only; per-trial provision (compose up + healthchecks) is paid at the top of each trial
Per-trial latency overhead None (containers already up) ~5–15 s for compose up + healthcheck on a task like coding_public_example_01; grows with the task's declared service count
Docker daemon load Constant Bounded by worker count × per-trial compose service count
Cross-trial concurrency ceiling One trial per stateful engine service (writes to the same DB) Bounded by host CPU / memory / daemon throughput, not by task shape

The A/B run above burned $0.225 total on shared (avg 44 s/trial) and $0.246 total on per_trial (avg 54 s/trial). Two effects to separate:

  1. Latency premium is real and reproducible. ~10 s per trial for a 3-service compose (up + healthchecks + gRPC connect + teardown). A second run of a domain-specific MCP-served task at higher turn cap saw a larger +25 s premium — the compose-up cost scales with service startup time (postgres warm-up, image cache state, healthcheck poll interval). Operators should budget per-trial isolation's wall-clock cost against expected trial duration.
  2. $-cost premium is not systematic at this sample size. The 3-trial per_trial arm's $0.021 total premium was concentrated on the cold-start trial 0 ($0.099 vs shared trial 0's $0.077, driven by a 2-turn-longer trajectory) — trials 1 and 2 costs matched shared within one cent. A follow-up run of CAP-001 in each mode saw per_trial come in cheaper than shared ($0.068 vs $0.079), which contradicts a systematic per-trial-mode $-overhead hypothesis. At this sample size, cost noise between LLM routing / token counts / trajectory length dominates any putative substrate-driven cost premium. See the follow-up in "Consequences → Follow-ups" for the measurement work needed to characterise this properly.

Tasks that declare many services or slow-starting services (large postgres seeds, image pulls) pay the compose-up latency each trial in per_trial. That translates to LLM cost only if the trial clock keeps ticking during compose-up in a way that consumes tokens — an open question flagged for follow-up measurement.

Failure-mode differences

Cross-referencing RUNTIME_BACKENDS.md's "Failure modes" section, the mode-specific soft failures are:

  • shared only — cross-trial state contamination. Two trials writing to the same DB can see each other's writes. Deterministic-grading tasks (whose grader reads the DB) can produce wrong verdicts if the code that writes to the DB doesn't scope by trial_id. Detection is task-author responsibility — the runtime doesn't know what a task's grader expects. Guardrail: per-service services.<name>.isolation labels drive backend selection. SharedStackRuntimeBackend runs a task only when every service is labelled isolation: shared — the task author's acknowledgment of the shared-state responsibility. A task declaring any service isolation: reset or ephemeral is auto-routed to PerTrialRuntimeBackend by task-driven selection. The deprecated orchestrator.runtime override is the only way to force a shared backend against a per-trial-requiring task set, and _verify_isolation_compatibility refuses that at startup.
  • per_trial only — daemon-throughput ceiling. Running many workers on one host multiplies compose-up cost by worker count. On daemons with low concurrent-project limits or slow image cache, workers can starve at the docker-daemon boundary. Guardrail: bound worker count against observed daemon throughput; ADR-0010 requires provisioners to fail loud on ProvisionError and attribute the failure deterministically so throughput ceilings surface in the aggregate report rather than looking like task failures.

Hard failures (image build error, network bring-up error, grader RPC error, runner crash) are shared between modes and covered in RUNTIME_BACKENDS.md.

Decision rubric — which mode when

A concrete flow, per task:

  1. Does the task declare any service isolation: reset or ephemeral? → the run lands on PerTrialRuntimeBackend, selected automatically. The declaration says the task's grader cannot tolerate shared state; task-driven selection routes it there without operator action.
  2. Does the task declare every service isolation: shared? → the run lands on SharedStackRuntimeBackend, which is safe and faster. PerTrialRuntimeBackend also provides stronger isolation than needed; the deprecated orchestrator.runtime override is the only way to force it.
  3. Does the task not declare an environment_manifest? → no services: map means nothing requires per-trial materialisation, so task-driven selection lands the run on SharedStackRuntimeBackend. PerTrialRuntimeBackend.provision() requires a manifest and fails loud without one, so a task that needs per-trial isolation must first author a manifest with per-service labels.
  4. Genuinely stateless workload where cost dominates? → label every service isolation: shared; the run lands on SharedStackRuntimeBackend (substrate materialised once per run), and the author owns the "no cross-trial contamination" invariant the label asserts.
  5. Cross-trial isolation matters more than cost? → label the relevant services isolation: reset or ephemeral; the run lands on PerTrialRuntimeBackend. Budget the ~10 s and ~9% cost premium per trial visible in the A/B numbers above.

Observability parity

Logging, metrics, and artifact writes do not vary by mode:

  • Structured logs use the same event names (Trial completed, Trial graded, Trial env teardown complete, Aggregate Results) with the same fields regardless of backend. The Provisioning trial env / Trial env provisioned events fire in both modes; on SharedStackRuntimeBackend they wrap a no-op that returns the shared endpoints.
  • Result aggregation writes the same TrialResult / Trajectory shapes to the artifact writer. Grades, costs, latencies, and termination reasons are all mode-agnostic fields.
  • TerminationReason.PROVISION_ERROR fires only on per_trial (the shared backend has no provisioning step that can fail per trial), so attribute_failure() classifying provision_failure is a mode-specific category — but the classifier itself is mode-blind, and the field is present in the aggregate for both modes.

Consequences

Positive

  • Operators pick a mode from a rubric, not from source-reading.
  • Grade equivalence is documented and empirically confirmed — future refactors that touch grading can be reviewed against this claim.
  • Failure-mode language is normalised; incident write-ups reference this section instead of re-explaining.

Negative / Trade-offs

  • The comparison table needs maintenance as backends evolve. Additions to a backend should refresh both this ADR and the side-by-side table in RUNTIME_BACKENDS.md.
  • Any future backend (Kubernetes, Modal, etc.) that doesn't fit the shared-vs-per-trial dichotomy will need this ADR extended or superseded.

Follow-ups

  • Task-pack migration to per-trial. Migration of the existing task packs that need per-trial isolation into environment_manifest-shaped tasks with isolation: reset / ephemeral service labels is separate task-pack work. This ADR does not gate that migration.
  • Runner image publication. RUNTIME_BACKENDS.md "Follow-up work" already tracks publishing engine images to a public registry so task compose files can reference published tags directly (:local stays as the local-dev alias). Independent of this ADR.
  • Per-trial concurrency benchmarks. The resource profile table gives order-of-magnitude framing; a proper benchmark harness would produce numbers on a range of task shapes and daemon configurations. Filed as a follow-up.
  • Cost-premium characterisation. The small-sample runs cited above don't distinguish LLM-routing noise from any putative substrate-driven $-cost premium. Open questions: does the trial clock consume LLM tokens during compose-up (e.g. via user-simulator warm-up or agent context establishment)? Does openrouter route consistently at temperature=0 across mode-differentiated runs? A ≥20-trial-per-arm run with token-level accounting would settle both. Filed as a follow-up.
  • Multi-service adapter compatibility. Some existing adapters materialise their own per-task compose stacks inside the tool-invocation path (see RUNTIME_BACKENDS.md "Adapter compatibility with per_trial") rather than through environment_manifest. Unifying those paths is future work.

Links

  • Related ADRs:
  • Related code:
    • tolokaforge/core/shared_stack_runtime.py
    • tolokaforge/core/per_trial_runtime.py
    • tolokaforge/core/trial_grader.py
    • tolokaforge/runner/service.py (GradeTrial handler)
    • tolokaforge/core/orchestrator.py — backend selection + isolation guard
  • Related docs:
    • docs/RUNTIME_BACKENDS.md — mechanics deep-dive
    • docs/ROADMAP.md