Skip to content

BACKLOG.md

Latest commit

 

History

History
173 lines (138 loc) · 48 KB

BACKLOG.md

File metadata and controls

173 lines (138 loc) · 48 KB
Raw

remote-toolkit — BACKLOG

tier: critical

One entry per ROOT CAUSE. Protocol: research-loop commands/CAPABILITIES.md § ISSUES protocol. Grep before adding (+1 <project> <date> on a hit); closes in the fix's own commit, replaced by a - [fixed <commit>] tombstone.

Sanitized text only. Host names, profile names, cluster paths and account names stay in the reporting project's own ISSUES.md.

Seeded 2026-08-01 from the second migration lane.

  • [open] id: remote-toolkit#first-apply-staging-enoent 2026-10-11 infra-agent: the first apply of a small changed file fails on the remote side with unable to set staged file permissions: ... chmod <beta staging path>: no such file or directory (a Transition problem), and the file lands only on the next cycle, ~30–60 s later — twice in one session, on a 9 KB and a 6 KB file of an active code-sync profile; meanwhile exec's flush gives up after 10 s ("may see slightly older files"), so a launch issued right after an edit can read the stale copy. Not the size cap of the 8dd1889 tombstone (KB-sized files). Worked around by comparing the remote digest before launching. → next: capture mutagen sync list -l at the moment of the failure; check whether anything prunes the beta staging directory between stage and apply. repro: edit a tracked file locally; within ~5 s run mutagen sync list -l --label-selector=rt-profile=<p> → Transition problems: <file>: unable to set staged file permissions ... no such file or directory; ~30 s later the file is on the remote with the new digest. +1 2026-10-12 infra-agent, twice more on the same profile, captured at the moment as the entry's next step asked: Transition problems: analysis/<file>.py: unable to set staged file permissions: unable to set permission bits: chmod /root/.mutagen/staging/<session>-beta/<2 hex>/<digest>: no such file or directory with Status: Applying changes. A MODIFIED file landed one cycle later; a NEW file (4 KB) had not landed after ~45 s and the session stayed in Applying changes — touch on the local file plus mutagen sync flush landed it within ~20 s. So the self-heal is not guaranteed; the digest check before launch stays the workaround. +1 2026-10-13 infra-agent (same profile): a MODIFIED 3 KB script stuck ≥ 4 min in Staging files on beta with the same chmod/ENOENT Transition problem; exec's flush, mutagen sync flush and a pause + resume all left it stuck (the beta staging dir <session>-beta/ existed and was empty, root disk 74 % used). It cleared only after the identical bytes were written to the beta path from another host (a pod on the same ceph), after which the session returned to Watching. So the self-heal is not guaranteed for modified files either; the digest check before use stays the workaround. 2026-10-14 sweep (Mutagen v0.18.1 source, no recurrence to observe): every Transition ends in stager.Finalize(), which RemoveAlls the endpoint's whole staging root, and that root is named ~/.mutagen/staging/<session-id>-beta — so only another process holding the SAME session id can delete a file this agent staged, and a stuck spell that survived pause + resume (a fresh agent) points at a second, actively driven agent for that session, not at a prune. Measured now: 8 synchronizer agents on the beta host, 8 local sessions to it, all started together (a reconnect ~22:58). wake-when: the failure recurs — at that moment count the beta host's mutagen-agent synchronizer processes and their ages against the Mac's sessions for that host; more than one agent for the session (or more agents than sessions) confirms the double-agent cause and points the fix at reconnect hygiene.

  • [open] id: remote-toolkit#push-drops-exec-bit 2026-10-10 fast-matmul-compiler: push installs the file with the remote shell's default mode (cat > tmp; mv -f tmp dst), so a local executable bit is silently dropped and a pushed helper that must be found on PATH (or run as ./x) fails with "Permission denied" while the push reported "verified". Worked around by a remote chmod 755 after the push and a wrapper that refuses a non-executable helper. → fix shape (a contract CHOICE, not taken here): carry the local mode bits (or at least u+x) to the installed file after the digest check, and say so in push's usage text. repro: chmod +x x.sh; rt -p <p> push x.sh x.sh; rt -p <p> exec --no-flush 'ls -l x.sh' → -rw-r--r--. +1 2026-10-10 14:46 fast-matmul-compiler, a worse shape: RE-pushing an UNCHANGED executable also rewrites it with the default mode, so a caller that re-stages before every launch (jobs/stage_intent.sh) made two live PATH helpers non-executable for ~16 s while a running job depended on them (no process hit the window). Caller-side mitigation shipped there (skip a push whose remote sha256 equals the local one; restore 755 on executables after the loop); the root fix here is unchanged. blocked-on: owner ruling (2026-10-14 sweep) — push documents only an atomic, sha256-verified install, so which mode the installed file gets is a contract choice: (a) cp/scp semantics — a replaced file keeps its own mode, a new file takes the local mode under the remote umask (recommended: the convention callers already assume, and it ends the re-push that stripped +x from live helpers); (b) always carry the local mode (rsync -p; can loosen a deliberately tighter remote mode); (c) only guarantee u+x when the source has it.

  • [fixed remote-toolkit@14bf380] id: remote-toolkit#new-profile-cannot-create-its-root 2026-09-30 attn-quant — a never-connected profile could not create its own REMOTE_DIR (every exec/fetch form cds there first; push never creates directories); SKILL.md now names the one-off bootstrap from another profile on the same host (exec --no-flush 'mkdir -p <root>'). The behavioural alternative (exec from the parent when the root is missing) was not taken; reconsider if a session hits this after the doc exists.

  • [fixed remote-toolkit@7142ac003002] id: remote-toolkit#bg-payload-expands-variables-early 2026-09-27 quant-robust: exec --bg expanded the payload's $var / $(...) in the login shell that parses the ssh argument, one layer before tmux started the job's shell — a silent wrong command. _exec_bg now builds the wrapper locally, base64s it, decodes it to ~/.rt_logs/<session>.sh on the host and has tmux's shell source it with ., so the payload runs byte for byte (which also ends the nested-quote mangling named in the start-verification tombstone); canary 1g (loop variable logs xa xb) and 1h ($(pwd), nested quotes, trailing comment) pin it and fail on the old rt; 219/219. +1 2026-09-27 quant-robust, second hit: a checkpoint-download loop fetched the default revision five times, exit 0.

  • [fixed remote-toolkit@430292d6890e] id: remote-toolkit#exec-wrapper-swallows-closing-paren 2026-10-01 attn-quant: plain exec and fetch wrapped the command as ( cmd ) on one line, so a command ending in a heredoc got the terminator line EOF ) and one ending in a # comment lost the ): the remote shell died on a syntax error before the trailer, and exec reported a 0-byte TRUNCATED stream. Both wrappers now put the command on its own lines inside the subshell; canary 17s / 17t / 18w fail on the old rt and pass on the new one (222/222); a live heredoc read returned its output, rc 0.

Extended 2026-08-02 by this tool's FIRST audit — a de-correlated read of rt, SKILL.md, CLAUDE.md and README.md against three questions (where is a state reported without being established · where can data be lost · which documented claims are enforced nowhere), with every finding reproduced against real Mutagen 0.18.1 on local-only endpoints or a stubbed SSH. The entries below are that audit's, not a project's; each says whether it was CONFIRMED offline or still needs a live host.

The lane has one shape. rt's callers are usually AGENTS, and almost every path that could say "I could not establish this" instead returns 0. A human notices an odd line; a bash guard branching on $? does not.

  • [fixed c4fe9d4] 2026-08-10 ceph-cleanup lane: exec --bg reported "Background job started" and exit 0 for a job the remote shell had rejected, while rt logs <session> answered "Log not found" — two commands disagreeing about existence, neither saying it failed to start. _exec_bg now probes tmux has-session OR the log file after launching and dies naming the job when neither is there; canary 1e/1f pin the refusal and its false-positive guard (1e fails with the check reverted). The payload mangling itself (nested quotes through three wrapping layers) is the encoding entry below, untouched: this closes the false statement, which is what made the mangling cost round trips. +1 2026-08-13 (single-line payload, nested quotes alone are enough). +1 2026-09-10 (fast-matmul WAKE 181): a $(sha256sum …) inside a single-line exec --bg payload came out as bash: syntax error near unexpected token '(' from the tmux wrapper; the reported-failure path now works (it said so), and the fix on the caller's side was the documented one — stage the body as a script and background that.

  • [fixed remote-toolkit@eef6d5d] 2026-08-07 audit: the OTHER half of the entry below — exec still hands a large payload back as a stream, and the cut is downstream of anything rt can reach. The tombstone below closes the silence: exec now certifies whether the stream it received was whole, and reports its size. It does NOT close the loss, and the measurement that decides this is in that tombstone: no consumer in this workspace branches on rt exec's exit status — all four real script callers pipe its stdout into tar/grep/awk, and a pipeline reports its LAST stage's status, so the new 125 is discarded before anyone sees it, and three of the four send stderr to /dev/null as well. The signal exists and the existing callers are deaf to it. ASK A HUMAN — this is a CONTRACT change, which is the line the routing protocol reserves: routing a payload above a threshold to a remote file and returning the PATH (plus a manifest digest) makes truncation structurally impossible instead of merely visible, but it changes what exec returns, so every caller above has to be migrated rather than merely recompiled. The question to put: do we add a separate verb (rt pull <remote-path> <local-path>, verified by digest, exec untouched) or change exec itself above a size threshold? A separate verb costs nothing to the four existing callers and can be adopted one at a time; changing exec is the one that ends the shape for good. The design note at the foot of this file already proposes the verified-manifest form and is the same decision seen from the input side. repro: rt exec 'cat <file larger than the caller's capture limit>' | tar xf - — the tar is short, and neither rt's new 125 nor its stderr line reaches the caller, because the pipeline swallowed both. +1 [2026-08-07 train-stab, INPUT side of the same contract]: base64 < local-file | rt exec 'base64 -d > remote-file' delivered an EMPTY stdin stream to the remote command in 2 of 3 invocations — base64 -d succeeded on zero bytes, wrote a 0-byte file, exit 0, follow-up echo printed; one invocation of the identical pipeline had worked minutes earlier, so the delivery is intermittent, not absent. Cost: a 0-byte python script 'deployed' cleanly and a GPU eval run consumed on python3 empty-file (rc=0, no output — indistinguishable from success until the artifact was checked). Workaround that ends the exposure: never pipe payload through exec stdin — stage via the profile's Mutagen dir, remote-side cp, verify sha256 both sides. Strengthens the case for the verified-manifest rt pull/rt push verb pair in the design note. +1 [2026-08-08 train-stab, same INPUT-side condition]: recurred twice in a row on a ~34KB payload (remote tar: gzip child status 1 = corrupted/truncated stream, not empty this time); cost compounded by a STALE-FILE MISDIRECTION — the target script existed from an earlier partial deploy, so the failed push left the OLD version in place and the next run produced a misleading old-format failure signature that consumed a debugging round. Mutagen-dir + remote-cp + md5-both-sides used for every deploy since; the workaround holds. [TRIAGE DO/L — harness session 28 cutover. Reviewed body sha 1511676deca9] The entry says ASK A HUMAN — this is a CONTRACT change. RULED HERE INSTEAD, 2026-08-07, with a de-correlated second opinion (gpt-pro, conversation 6a7544ed). rt exec still hands a large payload back as a stream and the cut is downstream of anything rt can reach; the sibling tombstone made truncation VISIBLE (exit 125) and measured why that is not enough — all four real callers pipe stdout into tar/grep/awk, a pipeline reports its LAST stage, and three of the four discard stderr too. The signal exists and every caller is deaf. The ruling: add the separate verb. Do NOT make execs return type depend on output size. The size-threshold option was the entry's 'ends the shape for good' candidate and it is the worse one — the semantic TYPE of the result would depend on a threshold (stdout below it, a path+manifest above), so every consumer implements two protocols, an implementation limit becomes public semantics, and a payload that grows silently changes the response shape with no caller change. That institutionalises the shape rather than ending it. And the correction that makes this L rather than S: adding the verb WITHOUT migrating the four callers does not fix the defect. One bounded change set — add the verified transfer primitive, migrate all four known large-output callers, test truncation / digest mismatch / interrupted transfer / destination atomicity, and KEEP exit 125 on exec as a diagnostic for unknown consumers. Note rt pull <remote> <local> may be too narrow if a caller GENERATES its payload rather than reading an existing file; the decision is a separate bulk-output contract, not that exact verb. pipefail is not a substitute — downstream stages may already have consumed partial bytes. A ruling would be needed only to RETIRE or redefine rt exec, which this does not. +1 2026-08-07 train-stab, the INPUT side, and it changes the deliverable: the verb must be a PAIR. base64 < local | rt exec 'base64 -d > remote' delivered an EMPTY stdin stream in 2 of 3 invocations — base64 -d succeeded on zero bytes, wrote a 0-byte file, exit 0. A 0-byte script 'deployed' cleanly and a GPU eval was consumed running python3 <empty file>, rc=0, no output, indistinguishable from success until the artifact was checked. Intermittent, not absent, which is why it survived. So the same contract is broken in BOTH directions and a pull-only verb closes half of it; the design note at the foot of that file already proposes the verified-manifest pair. Carry the interim workaround into the docs regardless of when the verb lands: never pipe payload through exec stdin — stage via the profile's Mutagen dir, remote cp, verify sha256 both sides [FIXED 2026-08-08, harness session 29 — the verb PAIR, and all known callers migrated.] rt fetch <local> '<remote-command>' and rt push <local> <remote>. The remote stages the payload, digests it THERE, streams it, and reports the digest in the same nonce trailer exec already uses; rt digests what arrived and renames into place only on a match. So a cut anywhere — remote, transport, or rt's own stdout — is a digest mismatch, and on every non-zero path the destination is not written and the incoming file is removed. Exit codes reuse this repo's existing vocabulary rather than inventing one: 0 verified · 2 unverifiable (no sha tool — never a mismatch, the distinction rt verify already draws) · 3 mismatch · 124 bound · 125 no outcome established · otherwise the remote command's own status. No pull verb, deliberately: one of the two real large-output callers GENERATES its payload with tar cf -, so a path-taking pull could not have served it — which is the narrowness the ruling warned about. cat <file> is just a command. The migration is the half that fixes the defect, and it is done. opt-dyn/evidence/sync_runs.sh had NO verification at all (exec 'tar cf -' | tar xf -) and mirrors the artifacts paper-trail's measures extract from; it now fetches to a temp and extracts from a verified file, and its remote command was changed to ls -1 … | tar cf - -T - because an unmatched glob made tar emit a PARTIAL archive that the old pipeline extracted happily. rl-sgd/evidence/sync_runs.sh had hand-rolled the same discipline with an md5 round trip per file; the pipe is gone and both its guards are kept and documented as answering different questions — fetch and md5 both answer "did these bytes arrive intact" (independent digests, so a defect in one cannot silence both), and neither answers "are these the bytes we want", which is what its never-shrink guard is for. The three remaining rt exec call sites in those scripts are ls, md5sum and a grep|sed reduction — line-count outputs, not bulk payloads. Migrating them would be motion, not coverage. exec's 125 is kept as a diagnostic for consumers that have not migrated, per the ruling. Tests: canary_rt.sh block 18, 16 checks — happy path, byte-exactness through fetch (NULs, no trailing newline), no-trailer, digest mismatch on a SAME-LENGTH corruption (a length check alone passes that), a non-zero remote command, no-sha-tool as 2-not-3, push's empty-stdin incident verbatim, an existing destination left untouched, no incoming files left behind on any path, and a false-positive guard proving the happy path still writes after every refusal above it. 178/178 (was 161). _sync_preflight_for_exec was EXTRACTED from cmd_exec rather than copied into fetch: it is a decision rule, and this workspace has already paid once for carrying one of those in two places.

  • [fixed remote-toolkit@69be333] 2026-08-06 train-stab: exec truncates a large remote stdout silently and still exits 0 — the lane's own shape, on the output path. A command whose stdout exceeds roughly 100KB comes back cut off, with no marker in the stream and no non-zero status, so a caller that pipes several files through one exec receives a partial payload that parses as a complete one. Measured: a multi-file pull returned 1 of 7 files and reported success; the loss was caught only because the caller happened to hold a manifest of what it had asked for. Same failure CONDITION as a cross-repo record from 2026-07-13 (rt exec losing 6 of 7 files): one pattern, two occurrences, three weeks apart. Closed as the entry itself split it — the additive half only. Where the cap is NOT, established by reading the path rather than by measuring a symptom: there is no cap in rt, and ssh does not truncate. _exec_sync inherited its stdout straight through. So the ~100KB cut is in whatever CAPTURES rt's stdout — an agent's tool-output limit, a harness buffer — and rt can neither see nor prevent it. That is why the fix is two assertions and not a bigger buffer: the remote shell now writes a TRAILER to stderr after the command exits (a per-invocation nonce token plus the command's status), so a stream that ends without it is reported output is TRUNCATED and exits 125 — "no outcome established", beside the 124 already used for the bound, and a real transport status is never replaced by it. And the payload is COUNTED on the way through, with a complete one at or above 64 KiB (RT_EXEC_REPORT_BYTES) reporting its size, which is the number a caller compares against what it actually holds. Below the threshold nothing is printed: a small output behaves exactly as before. stdout stays a byte-exact pass-through, and that is load-bearing rather than tidy — a live caller pipes tar cf - through exec into tar xf -, so the trailer had to go on stderr (filtered back out by a nonce-matching awk) and the count had to be taken by tee/wc over a fifo rather than by any line-oriented filter. Pinned by a NUL-byte, no-trailing-newline payload. Consumers counted before touching the status set (rg --no-ignore, 2026-08-07): 405 static occurrences in 168 files across 83 directories, but that is the MENTION count and it is the wrong predicate — the decision changes an exit status, so the predicate is who branches on it. Excluding prose, notes and captured transcripts leaves four real script callers, and not one branches on rt exec's status: every one pipes stdout into tar/grep/awk, which discards it. So 125 breaks nobody, and — the half worth writing down — helps nobody either without the follow-up entry above. Two defects in this fix were found by attacking it, not by the suite, and both are the shape the fix is about. (1) Waiting for the accounting readers reinstated an UNBOUNDED wait: they see EOF only when every holder of the payload pipe closes it, which after --timeout kills ssh can include something ssh left behind — a 1s bound returned after 4s. _exec_reap bounds that wait at 2s and then kills, and because wc writes its total only at EOF the count is then reported as unestablished, never as 0. (2) The trailer proves the COMMAND wrote everything it had and says nothing about whether rt handed it all on: exec 'cat 400KB' | head -c 5 reported "81920 bytes of stdout, complete" — the same unsupported benign claim, reintroduced one layer out. The pass-through's own status is now a third input, and a consumer that closed early is reported as a PREFIX with a non-zero status even when the command exited 0. Note the boundary that check had to find: at 70 KB the defect does not reproduce, because every byte still reaches rt's stdout and the kernel buffers it — the claim only becomes rt's to get wrong once forwarding itself cannot finish. A third, smaller correction came from a mutation run on this commit's own comment: the wrapper's trailing exit $? was described as what keeps exec's status the command's, and it is not — the status is read out of the trailer, and removing that line left the suite green. It is kept for the narrower reason now written beside it. Tests: 19 checks (block 17), mutation-scored in both directions — reverting _exec_sync fails 4, dropping the subshell that protects the trailer fails 1, leaking the trailer into stderr fails 1, eating the partial stderr line fails 1, ignoring the forwarding status fails 1, always claiming it fails 4, reporting on every output fails 1, reporting an uncounted stream as 0 fails 1. Suite 142/142 before, 161/161 after. The pre-existing --timeout check was widened (stub sleep 4→12s, threshold 4→6s) because the reap grace left it no margin on either side — fix-induced, and it flaked once under load before the change

  • [fixed 8dd1889] 2026-08-12 ceph-cleanup lane: erroring could not distinguish a condition that will clear from one that never can, and status --all listed a dead profile among the healthy ones in the same visual form — measured at two days without a single completed synchronisation, reading to a session as "currently retrying". SuccessfulCycles was in the record rt already fetches and was being discarded; zero of them is exactly "has never once succeeded", which is the fact the state word cannot carry. A non-synchronising row is now marked, the never-completed fact is named when the count is established and zero, and the summary counts those separately from the live sessions. Not established stays non-zero and prints nothing — a count that could not be read must never render as "never synchronised". Verified on this machine's real sessions: 10 profiles, 2 never once synchronised, previously indistinguishable. Elapsed-time escalation, this entry's other fix shape, was NOT built and is not owed: it needed new persisted state to answer, by proxy, the question the cycle count answers directly and for free. Canary 180→194; 4 mutants. Body → git history.

  • [fixed 8dd1889] 2026-08-12 ceph-cleanup lane: a file above MUTAGEN_MAX_STAGING_FILE_SIZE was fetched in full, discarded, and reported as a chmod/ENOENT on a content-addressed staging path that never mentioned the cap — measured at two files (1.34 GB, 820 MB) against a 256 MB cap re-fetching ~2.1 GB per cycle forever, reading as a corrupted staging area. _sync_error_hint names the limit behind the failure shape at all three surfaces that print the raw error, covering the entry-count breaker in the same pass (status had never named it either). Offered as a SHAPE match and it says so: the staging path names no file, so rt has measured nothing, and a hint that overstated itself would send the next operator to raise a cap that was not the cause. The entry's "better" shape — a pre-flight scan at connect that refuses or warns on oversize entries — was NOT built: refuse-vs-warn on a live connect path is a behaviour CHOICE, and the diagnostic closes the measured harm (going around the tool to diagnose) without it. 3 mutants. Body → git history.

  • [fixed 13f9815] 2026-08-12 ceph-cleanup lane: a sync whose remote root had been deleted wholesale still reported active (measured: ~563 GiB / 192k files deleted out-of-band over 12 hours; the profile read active throughout). Ruled 2026-09-05 (Shuyang): the REPORT half only — rt status prints Sync root: ABSENT locally / on the remote and status --all tags the row ⚠ LOCAL/REMOTE sync root ABSENT (<dir>) and counts it in the summary; one stat per endpoint, the remote one only while Mutagen reports the beta endpoint connected (through the profile's own ssh identity), and a host that does not answer is (remote root not established), never absent. Propagation untouched, and the line says so. Canary 19a-19j; mutants: a non-zero ssh read as absent (19d), the connected-endpoint gate dropped (19f). The behaviour half — which way Mutagen resolves an absent root — stays a human decision at the moment it arises; the report exists so that moment is seen. Body (two fix shapes recorded as NOT built, the 2026-09-04 recommendation) → git history.

  • [fixed 13f9815] 2026-08-13 grpo-speed s53: stopping a foreground rt exec from the local side killed only the local ssh, the remote command ran on as an orphan, and nothing said so (six orphans found by hand). Ruled 2026-09-05 (Shuyang): exec --bg is the one stoppable lane by design; plain exec gains the honest half — _ssh_bounded carries the unbounded call too, watched for INT/TERM/HUP: the handler sets a flag, the loop (or an early-returning wait) kills the ssh as a tree and reports Stopped from this side (SIGx): … does NOT stop the REMOTE command, exits 128+signal, and the accounting marks what arrived as a PREFIX cut by the stop (decided by the flag, never by a status ≥128 — a remote command killed on the host comes back with the same numbers). Never fires on a run that ends on its own. SKILL/README name the lane. Canary 14j-14m; mutants: traps removed (14j), stop branch dropped from the accounting (14l), kill-after-wait ordering (14k — found live: 13s on a 12s stub, i.e. no stop at all). Body + the 2026-09-05 sweep analysis → git history.

  • [fixed remote-toolkit@67a8df8] 2026-08-01 dllm-reasoning: nothing told a caller when a staged file had actually ARRIVED, so callers waited a fixed sleep and then installed a stale copy. Measured: a patched launcher script was staged, waited on for ten seconds, and the still-stale remote copy was copied into place — bash -n passed because the OLD file is valid shell, so the run started and died instantly on an unknown case, holding 8 GPUs idle for ~25 minutes. [RULED 2026-08-05, human, decisions §5.3] THE ARRIVAL ASSERTION ONLY — the per-run staging wrapper stays a separate project (register §3.3), and this fix does not touch it. rt verify [--timeout N] [--interval N] <path>... hashes the named paths locally, has the remote hash the same paths, and compares whole manifests, polling until they agree. The local side is recomputed AFTER the remote answer, so a file edited during the check is not certified — the contract is "equality was observed and held across the check", not "these bytes are frozen", which nothing here could promise. Three exit codes, the same split the flush codes use: 0 equality observed, 2 blocked or unverifiable, 3 the deadline elapsed and the sides still differ — and 3 deliberately does NOT say "still syncing", because a mismatch is not evidence of progress. Everything that can never be verified is decided BEFORE any waiting and exits 2: an absolute path, a traversal, a directory, a symlink, a path absent locally, a path matching the LIVE session's ignore rules (it can never arrive by syncing, so waiting for it waits forever), a sync session that is not propagating, and a missing sha256 tool on either side — the last being unverifiable, never a mismatch and never an arrival. The ignore matcher implements only the pattern shapes rt itself generates, so a MATCH is authoritative and a NON-match claims nothing; reimplementing Mutagen's full grammar in Bash would produce a checker that is confidently wrong. Verified end-to-end against a real remote: staged edit → rc 3 naming the differing file with both digests → wait → rc 0 → remote confirmed to hold the new bytes

  • [fixed remote-toolkit@67a8df8] 2026-08-02 audit: "flushed" did not mean the endpoints were equal, and nothing in rt established that they were. Mutagen's flush forces a synchronization CYCLE; it does not prove byte-for-byte equality. CONFIRMED with the real engine: two-way-safe with an unresolved conflict, and an ignored path, both flush with exit 0 while the endpoints differ. Closed with the entry above, which it explicitly rides on: the DOC half now says plainly in both SKILL.md and README.md that a returned flush means "a cycle completed", not "the remote has your file", and points at rt verify as the assertion a flush is not

  • [fixed remote-toolkit@9f58754] 2026-08-01 reason-select: a courier bucket degraded silently as finished handoffs accumulated, and the tool's only lever made it worse. Measured: 10,292 files / 2.6 GB in the tree put a flush in Scanning files for 20+ minutes — a 9 KB push took ~40 minutes and a 2.7 MB pull had not arrived after 25 — while the same bucket at 232 files / 41 MB flushed in 7 seconds. Everything in it was FINISHED pulls nobody cleared. _sync_flush now reports the alpha endpoint's file count and byte size before flushing, defaulting to a warning above 2000 files or 1 GiB (between the two measured points, leaning early since the whole point is to speak before the stall); rt status prints the size on every call. The message carries the standing lesson, because the intuitive fix is the wrong one: PRUNE the bucket — a growing ignore list is the tell that it is being used as storage, and ignores do not take effect until the session is recreated anyway. Two things the size check must not do, both pinned by tests: it must not treat an UNSCANNED endpoint's zeros as a measurement (a breaker-halted session reports Scanned: false with 0|0|0, and _sync_counts refuses rather than answering), and it must not print on every rt exec — it is rate-limited to once per hour per profile, because a warning printed hundreds of times is one that gets filtered, and this workspace already has a caller grepping ^!! Sync flush out of its output. The counts ride in the existing status probe; probe cost was measured before assuming it mattered (11 ms against 438 ms for a trivial rt exec) +1 2026-09-15 research-project wrap: the same large-tree scan condition recurred at 91,745 files / 6.8 GB; a staged cell was still absent after roughly 30 minutes although sync was active. The verified rt push route completed the needed small transfers. This instance contains durable experiment evidence rather than disposable completed courier files, so the old PRUNE recommendation cannot be applied until the project owner selects the artifact-storage policy. No sync root, ignore rule or evidence file was changed for this routing update; no costly live scan was repeated. doctrine-grep: carried — SKILL.md arrival section says flush/active status is not byte equality and directs callers to rt verify; its payload section says push already bypasses Mutagen and needs no --no-flush flag. Existing diagnostics/workaround reached the incident; the remaining blocker is the project owner’s git-versus-remote-artifact storage policy, not an unimplemented transfer primitive. This +1 preserves the tombstone’s diagnostic scope and does not claim scan latency was fixed.

  • [fixed remote-toolkit@fc02cb5] 2026-08-01 grpo-speed: bounded reads through exec failed on macOS with "command not found: timeout" — the GNU binary is not present on darwin (it is gtimeout, from coreutils) and callers wrote timeout N rt … exec …. Fixed by implementing the bound IN THE WRAPPER rather than detecting gtimeout: rt exec --timeout SECS is pure Bash, needs no GNU binary and no PATH assumption, and exits 124 so a caller that already branched on timeout(1)'s status does not have to learn a new code. Documented at the bounded-read path in both SKILL.md and README.md, including the nastier sibling form recorded by another project — timeout 120 <cmd> | head -3 exits 0 because head succeeded, so a read that NEVER RAN is indistinguishable from a successful read that returned nothing. The bound is on the WAIT, not the work, and says so when it fires: killing the local ssh does not reliably kill the remote process. A live test caught a real bug in the first implementation: backgrounding _ssh (a shell FUNCTION) makes $! the subshell's pid, so kill left the real ssh running — the bound returned at 3s while the remote command's output still arrived at 30s, because the surviving ssh held the pipe open. ssh is now invoked directly. --timeout with --bg is refused, since a background job outlives the shell by design

From the 2026-08-02 first audit

  • [fixed remote-toolkit@b9e35f8] 2026-08-02 a background job's recorded exit status was tee's, not the command's. { cmd; } 2>&1 | tee log; echo "EXIT_CODE=$?" records the LAST element of the pipeline, and tee essentially always succeeds — so every job that ever started was recorded EXIT_CODE=0 and listed by rt logs as [DONE], including one that died on its first line. A caller reading that list to decide whether a run finished was told yes, always. Fixed with a POSIX marker file (${PIPESTATUS[0]} is bash-only and tmux starts the remote LOGIN shell, so on a dash host it would expand to nothing — the same class of silent failure). The command runs in an explicit subshell so that a literal exit N cannot terminate the pipeline's own subshell before the marker is written. rt got its first tests with this: test/canary_rt.sh sources rt, stubs _ssh, and RUNS the captured remote command locally, so the four failure shapes are pinned by behaviour rather than by wording

  • [fixed remote-toolkit@4a4b376] 2026-08-02 a failed flush was converted into a successful CLI call — and the reason the tool held two OPPOSITE policies for it (exec ran anyway, slurm submit aborted) is that "flush failed" was one word for two states. _sync_flush now returns 2 (paused / mutagen error — nothing synchronised) or 3 (the time bound elapsed; the daemon is still working). exec refuses on 2 and continues loudly on 3, because refusing on a routine timeout would make --no-flush habitual and turn the protection off entirely; slurm submit refuses on both, since a queued job outlives the shell; sync flush exits non-zero. Human ruling 2026-08-02 — the two states had to be separated before any single policy could be right

  • [fixed remote-toolkit@1a309cf,36adcb5] 2026-08-02 an unhealthy sync session was classified active. The classifier had no negative case: anything carrying Identifier: that did not match Status:.*Paused and did not match Connected: No was called healthy, so every state Mutagen has that those two strings do not name reported as healthy. The one that actually happens is the entry-count circuit breaker, which reports Connected: Yes on BOTH endpoints with Status: Waiting … for rescan. Re-confirmed against real Mutagen 0.18.1 2026-08-05. _sync_status now classifies from a structured mutagen sync list --template probe of .Paused / .Status / .LastError / .Alpha.Connected / .Beta.Connected, and the classification is POSITIVE: active requires a status verb on a recognised-healthy list, so a verb this tool does not know (a future Mutagen adding a brake) reports unknown loudly instead of rejoining active silently. Seven states rather than four; the new ones are erroring (a live error — measured NOT sticky, it clears within one rescan interval once the condition clears, so it is a live condition and not a scar; the entry-count breaker lives here), halted (one of Mutagen's three SAFETY BRAKES) and unknown (the daemon could not be asked, or two sessions matched one profile's labels — previously encoded as none, i.e. "there is no sync session", a claim about the remote that nothing established). The first version of this fix keyed on .LastError alone and an independent review refuted it: measured 2026-08-05 by driving each condition, all three of HaltedOnRootEmptied / HaltedOnRootDeletion / HaltedOnRootTypeChange report an EMPTY .LastError with both endpoints Connected: true, so an error-only classifier called all three healthy — the same defect this entry is about, one state over. The entry's own fix shape said "classify on Last error: and on the status verb"; only half of it had been implemented. halted and erroring are separate states because the REMEDY differs: an erroring session is fixed by repairing the ignore list and recreating, while recreating a halted one would silently decide which side's contents win, so its message points at mutagen sync reset instead. Two consequences were larger than the classification itself: (a) measured 2026-08-05, a flush against a halted session does not fail, it HANGS — still running after 15s — so it used to burn RT_FLUSH_TIMEOUT and return 3, "the daemon is still syncing", which exec continues on; _sync_flush now returns 2 for halted, erroring and unknown. (b) connect says plainly that it cannot recover either non-converging state instead of resuming a no-op and printing "Already connected" — resume is for PAUSED sessions. connect also refuses to create a session while the daemon is unaskable, which could otherwise have made a second session for a profile that already had one. A THIRD fail-open was found by the same review, in the consumer rather than the classifier: cmd_exec gated its whole flush block on _has_sync, a BOOLEAN that cannot carry "I could not find out", so an unaskable daemon skipped the flush and ran the command with no warning — the most permissive path taken for the least established state. exec now refuses on unknown and names --no-flush as the deliberate escape. SKILL.md's claim that a breaker halt shows as sync=paused was false and is corrected — that recovery path never fired

  • [fixed remote-toolkit@7b6a027] 2026-08-02 disconnect reported success when termination failed, and deleted the local record either way. mutagen sync terminate … || true discarded every failure, then the state directory was removed and Disconnected printed with rc 0 — so a session that was still running, still propagating deletions, lost its only local record and the caller was told the job was done. _sync_terminate now returns mutagen's own status and echoes its message; clear_state runs only after termination is ESTABLISHED, and "established" means re-asking the daemon rather than trusting the exit code of the command that was supposed to do it — a terminate that reports success while the session survives is also a failure. unknown (daemon unaskable) does not license deleting the record either: it cannot terminate a session it cannot see. Every refusal says the state was NOT cleared, so the operator knows the record survived

  • [fixed remote-toolkit@4a4b376] 2026-08-02 disconnect killed every background job while its own documentation said it preserved work — two intentions under one name, with no flush before or after, so a running job was interrupted and its last output could never reach local. Default is now the documented behaviour and it SAYS how many it leaves running; --kill-jobs is the deliberate form and flushes first. clear_state also stopped deleting the Slurm submission history, which is the only local record of what was queued and has nothing to do with whether a sync session exists. Human ruling 2026-08-02: separate the two intentions, default to the safe one

  • [fixed remote-toolkit@71c815b] 2026-08-02 agent-supplied values reached the remote shell unquoted. slurm cancel interpolated its arguments straight into scancel $*, so rt slurm cancel "123; <command>" constructed a remote compound command; --name and a log ID likewise entered unquoted positions, and a REMOTE_DIR or script path containing an apostrophe broke the ad-hoc single quoting. Two controls, because one does not cover every site: _shq (POSIX single-quote encoding, verified to round-trip every metacharacter under dash as well as bash — the remote runs a login shell) is applied at all eleven interpolation sites, and a GRAMMAR (_require_jobid accepting 12345 / 12345_7 / 12345_[1-9], _require_jobname accepting [a-zA-Z0-9][a-zA-Z0-9._-]*) guards the values that also flow into the exec --bg payload, where correct quoting through three nested layers is more fragile than refusing the characters. A REMOTE_DIR containing an apostrophe is now REFUSED at load_config with a clear message rather than silently building a malformed remote command, since that one value cannot be carried through the --bg payload at all. Tests RUN the captured command against stub scancel/sbatch binaries and assert the resulting argv, so they pin what the remote shell would do rather than how the source reads. Three test-design errors were caught by mutation runs and are recorded because each is the same trap: the first REMOTE_DIR check tested $(…), which '…' already blocked, so it passed with the fix reverted; _ssh's capture-to-variable stub read empty inside _slurm_submit's command substitution, which would have read as "nothing was sent"; and the log-id check was passing because load_config died on an empty RT_CONF before ever reaching the validator — sourcing rt never runs _init_profile

  • [fixed remote-toolkit@d2305f8] 2026-08-02 status --all enumerated state directories, not configured profiles, so a configured-but-never-connected profile was invisible — exactly when you would run it to find out what exists, and the remote SKILL's first instruction is to run this command to enumerate profiles. Now walks <host>/*.conf and unions in any state dir that has no config (a profile whose .conf was deleted while connected is still syncing; dropping it would reproduce this defect one directory over), marks that case, and prints the count and the directory it looked in — a bare row list cannot say whether "not listed" means "does not exist" or "not looked for". A directory that is not a valid host-group segment (_archive/) is skipped by the same rule that validates -p, not by a name list. Config identity is read in a SUBSHELL: configs are sourced bash, so reading eleven in one shell would leak each one's vars into the next and a profile that omits REMOTE_DIR would display the previous profile's. NUMBER CORRECTION: the 2026-08-02 record said "11 profile config files, 4 shown". Re-derived 2026-08-05 on the same machine: 11 configs, 10 state dirs, 1 invisible — the defect reproduces, the magnitude does not

  • [fixed remote-toolkit@72d44fa] 2026-08-02 documented sync scope was materially wider than the implementation. The defaults ignore outputs/, checkpoints/, wandb/ and others; the docs said files sync to and from the remote and never enumerated the exclusions, so a result written to any of those paths never became part of the local replica and nothing said why. The default set is now enumerated in both SKILL.md and README.md, next to the three consequences that cost the time — and a canary iterates the code's own RT_DEFAULT_IGNORES array against both documents, so a pattern added to rt alone now FAILS the suite rather than silently widening the gap this entry is about (with a positive fixture proving that check can still fire). rt sync status prints the live session's effective ignores via --template, not the profile's current text, and — the case that actually bites — DIFFS the two and names the patterns that exist in only one, because Mutagen freezes ignores at session-CREATE time, so editing MUTAGEN_IGNORE changes nothing about a running session while making the config file wrong about what is excluded. With no live session the list is labelled as what the next connect WOULD apply, since calling it "effective" would claim a session state that does not exist. _sync_create and the drift check share one _ignore_patterns producer so the two cannot disagree about what the config means

  • [fixed remote-toolkit@30e180c] 2026-08-30 staging a source TREE in a shared scratch profile trips the per-session entry breaker, and the docs only described the runaway case. The --max-entry-count paragraph told you to fix MUTAGEN_IGNORE and recreate — but a 1522-file repo working copy is not junk to ignore, so that recipe has no move. Measured: scratch-sh sat at 2724 entries (693 dirs + 2031 files) against a 5000 cap; a 27 MB tree took it over, Last error: alpha scan error: exceeded allowed entry count, and the session halted for every consumer of that profile. Deleting the directory did NOT clear the halt; rt disconnect && rt connect did. SKILL.md §4 now carries the positive form: tar the tree, sync the single file, extract on the remote OUTSIDE the synced directory

The design note this lane produced

The standing claim in the harness register — "Mutagen is being used as a job-control protocol: eventual bidirectional sync decides which code a GPU job launches and which result is authoritative" — was put to the de-correlated reader as a question, and came back TRUE with one qualification: ignored result paths are remote-only rather than bidirectionally authoritative, and under the default mode the LOCAL side wins every conflict, so a local edit can overwrite a remote result regardless of which was written later.

Its recommended smallest incision, recorded here rather than acted on: a per-run staging wrapper around exec --bg and slurm submit — copy a deterministic input snapshot to a unique remote run directory, verify its manifest digest, launch THERE, write outputs there, and fetch them with a verified manifest. Mutagen stays a browsing convenience and stops being the correctness boundary. That is a design change to a shared tool and belongs to a human.

  • [fixed remote-toolkit@bb2d45dd3e41] 2026-09-08 fast-matmul-compiler: verified single-file transfer lacked a sync-independent download path. Owner approved 2026-09-12: add rt fetch --no-flush using the shared exec preflight override, keeping default flush behavior, SSH reachability, SHA-256 verification and failure-without-destination-write semantics; push already bypasses sync. CLI help, README and SKILL expose both routes; offline canaries exercise default flush, paused/unknown overrides, SSH refusal, changed bytes and remote-command failure.
  • [fixed remote-toolkit@b120bc5102ef] 2026-09-27 research project: rt push into a remote directory that did not exist yet reported only "push FAILED (status 98)"; the remote's own RT_NO_DEST_DIR line goes to stderr and is easily cut by a caller's | tail -1, so four pushes read as an unexplained failure. The status-98 arm now says the parent directory is missing, that push never creates directories, and to mkdir -p first. No contract change (push still refuses to create directories); canaries 217/217.