Model registry cleanup + engine reliability fixes - #88
Open
jeffhamons wants to merge 5 commits into
Open
Conversation
OpenCode keeps its session store at $XDG_DATA_HOME/opencode/opencode.db, defaulting to ~/.local/share/opencode -- one file, shared by every worker, opened for write, growing without bound. On 2026-07-27 it stood at 1.78 GB and two of three concurrently-launched workers died before their first tool call with "database is locked", recorded as model failures though no model had run. Because every non-Codex model routes through this engine, the contention penalises exactly the cheap tier, and worsens with parallelism. Point XDG_DATA_HOME at a per-run directory under the existing scratch root so each worker gets a private store. Credentials live beside the DB, so seed the fresh root by COPYING auth.json -- never symlinking, since the Seatbelt profile denies writes outside SCRATCH and a symlink would resolve straight back to the shared file this exists to avoid. Also add OC_BASE ($HOME/.opencode) to the profile's allowed subpaths, and note on the --no-sandbox path that it still shares the global DB: fine for a lone full-access task, not for a fan-out.
…bris Lane checks that run pytest were recording FAIL in ringer runs but passing on manual reruns of the same verify command. Traced to fix-swarm.py's unconditional `git add -A`: it stages ANY file present in the task worktree, not just the agent's actual diff, so the owned-files gate then flags incidental debris as an unauthorized change and fails the whole check even though verify (pytest) passed. Confirmed two debris sources via reproduction against the real 2026-07-19 sprint-wave-3/4 task worktrees: - w3-4598-wallclock: fresh pytest-of-<user>/ basetemp junk. The worker engine (opencode) runs pytest under a sandbox that returns EPERM on writes to ~/.jeff-os/_test_isolation/ and the system temp dirs; Python's tempfile falls through to cwd (the task worktree) as a last resort, leaving pytest-of-.../pytest-N/ debris behind. - w4-2369-refresh: a stale, unrelated .jeff-os/state/argus/checkpoint.json dated May 14 -- 2+ months old, sitting untracked in the worktree from setup, first swept in by this run's git add -A. Fix: unless owned-files is the wildcard "*", stage with `git add -u` (tracked-file modifications only, so an out-of-lane edit to an existing file still trips outside_owned_files) plus `git add -- <owned paths that exist>` for new files under the declared lane. Untracked debris outside the lane is never staged, so it's never flagged. Verified against both task worktrees: checks now PASS with the debris still present, and a synthetic out-of-lane edit to a tracked file still correctly fails outside_owned_files. Ringer's own check-subprocess env (_run_check, ringer.py:6766) was investigated and ruled out -- not the cause.
A Grok Build 402 "usage balance exhausted" incident caused every task on that engine to burn both retry attempts (~5s each) before failing, and all of it landed in the scoreboard as ordinary model failures. - detect_engine_down_reason() classifies worker output (tail-scoped, ~2000 chars) for billing/auth terminal patterns: HTTP 402, "payment required", "balance exhausted", insufficient credits/quota, expired auth/token/ session/api-key, invalid api key. - RingerRunner tracks which engines are down for the run; once one task reveals it, every other task on that engine (running or queued) fails fast with status "engine-down" / verdict ENGINE_DOWN instead of burning a second attempt or a fresh taskdir. - aggregate_model_log_rows / aggregate_model_scoreboard_rows now exclude ENGINE_DOWN-verdict tasks entirely, so a billing outage no longer drags down a model's pass_rate/first_try_pass_rate. - engines/mock_worker.py gains a MOCK_ENGINE_DOWN directive for testing. Deviation from the "verification executes the artifact" invariant: the engine-down path skips running the check entirely (nothing to verify from a billing/auth failure, and PASS is never claimed). Stdin-closed, explicit sandbox mode, and "logs carry raw worker output" are unaffected — the new [ringer.py]-prefixed log lines follow the same convention as existing attempt-lifecycle lines. 🤖 Generated by JeffOS
Follows docs/TAXONOMY.md identity procedure for slugs the scoreboard flagged unregistered: - grok-4.5 (grok engine): the manifest "model" field grok-engine tasks actually log is "grok-4.5", not the existing "grok-build" default_model_key entry. Adds a parallel registry key resolving to the same xAI artifact. - 13 OpenCode/OpenRouter slugs (Anthropic, Cohere, DeepSeek, InclusionAI/Ant Group, Moonshot AI, NVIDIA, Alibaba/Qwen, Thinking Machines Lab), matched against ./ringer.py catalog and cross-checked against each lab's own page where the org segment wasn't self-evident (grok-4.5, kimi-k2.6, ling-3.0-flash, inkling). Left openrouter/nousresearch/hermes-3-llama-3.1-405b:free and openrouter/poolside/laguna-m.1:free unregistered — neither slug exists in the current OpenRouter catalog (catalog has the 405b model without a :free variant, and laguna-s-2.1/laguna-xs-2.1 but no laguna-m.1), so registering them would fabricate identity. Likely a stale/typo'd manifest slug worth a separate look.
Both remaining "unregistered" slugs traced clean via ~/.ringer/openrouter-catalog.changes.jsonl added/removed events — neither was a manifest typo: - openrouter/poolside/laguna-m.1:free existed in the OpenRouter catalog from 2026-07-09 until it was delisted 2026-08-01T23:13:22Z, hours before the prior commit's catalog check ran. - openrouter/nousresearch/hermes-3-llama-3.1-405b:free existed from 2026-07-09 until its free tier was delisted 2026-07-19 (the paid variant is still live). Registers both so historical attempts resolve to their correct lab (Poolside, Nous Research) even though the models are no longer orderable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
grok-4.5(the actual manifest slug grok-engine tasks log, distinct from the existinggrok-buildkey), 13 OpenCode/OpenRouter slugs cross-checked against each lab's own page or OpenRouter catalog history, and 2 slugs that turned out to be legitimately delisted (not typos) —poolside/laguna-m.1:free(delisted 2026-08-01) andnousresearch/hermes-3-llama-3.1-405b:free(free tier delisted 2026-07-19), traced viaopenrouter-catalog.changes.jsonladded/removed events.git add -Awas sweeping sandbox/tempfile debris into the owned-files gate and failing checks that had actually passed verify; switched togit add -u+ explicit owned paths.Test plan
pytest tests/test_identity_evidence.py tests/test_taxonomy.py tests/test_signal_contract.py— 18 passed./ringer.py models --notes-file ~/.ringer/MODEL-NOTES.md— unregistered-slug warning cleared (was 16, now 0)tests/test_engine_down.py, existing fix-swarm suite)