Skip to content

Fix: launchd re-pin needs bootout, and never fake a successful start - #6401

Merged
vivekchand merged 2 commits into
mainfrom
fix/launchd-repin-and-bootout
Oct 14, 2026
Merged

vivekchand merged 2 commits into
mainfrom
fix/launchd-repin-and-bootout

Conversation

@vivekchand

Copy link
Copy Markdown
Owner

The bug

The sync daemon's LaunchAgent pins an absolute interpreter path at install time (python = sys.executable). When that interpreter stops being able to import clawmetry, the running daemon survives, because Python holds imported modules in memory. It dies at the next restart, and every relaunch after that fails instantly:

/usr/local/bin/python3: Error while finding module specification for
'clawmetry.sync' (ModuleNotFoundError: No module named 'clawmetry')

The documented repair, clawmetry sync --restart, could not fix it. _register_launchd wrote the corrected plist but never booted the job out first:

  • launchctl bootstrap refuses a label that is already registered
  • the legacy load -w fallback also fails, while still exiting 0, so no return code could see it

So the corrected plist landed on disk, launchd kept running the definition it already held in memory, and the command printed Running in the background. Your data is syncing to the cloud. over a crash-looping daemon.

Reproduced against a real registered job before changing any code:

$ launchctl bootstrap gui/501 ~/Library/LaunchAgents/com.clawmetry.sync.plist
Bootstrap failed: 5: Input/output error          # rc=5
$ launchctl load -w ~/Library/LaunchAgents/com.clawmetry.sync.plist
Load failed: 5: Input/output error               # rc=0  <- the lie

Found on a founder node on 2026-10-14. The daemon had been dead about five hours, launchctl print showed runs = 12, last exit code = 1, and the stale ~/.clawmetry/local_query.json still advertised a dead pid and a refused port. Assistant, Setup and every other view that reads through the daemon rendered an empty state with no indication why, while the sync strip sat on Step: starting.

The fix

  • bootout before bootstrap. The only sequence that makes launchd re-read the file, and therefore the only one that can re-pin a stale interpreter.
  • Retry the bootstrap briefly, since bootout is asynchronous and bootstrapping too eagerly loses the race with teardown.
  • Confirm the legacy path with a real lookup (launchctl print) instead of a return code that lies.
  • Stop faking success. If the job did not register, clawmetry sync --restart says so, points at the log, and exits non-zero.
  • The dashboard registrar had the same defect with kickstart -k as its fallback, which restarts the definition launchd already holds and so can never re-pin either. It now shares the corrected helper.

The NemoClaw sandbox registrar already did this correctly (Unload first in case it was already registered) and is unchanged.

Verification

  • 9 new tests in tests/test_launchd_repin_requires_bootout.py: call ordering, absence of kickstart as a fallback, the untrustworthy load -w exit code, and both success and failure reporting.
  • They were run against a simulation of the old behaviour first, and 3 of them failed there, so they catch the bug rather than describe the fix.
  • 105 pass across test_daemon_registration, test_launchd_*, test_status_daemon_truth, test_daemon_spawn_pid_lock, test_onboard_local_default, test_desktop_supervisor_resilience, test_cloud_cta_oauth, test_login_bare_namespace.
  • Passes on Python 3.9 as well as 3.11 (the lint job's floor).
  • Ruff on the changed file: 404 findings, down from 405 on origin/main, no new rule classes.
  • Wired into the lint job, so the ratchet counts it as listed (375 listed, was 374).

Known limits

This makes the repair work and makes a failed start visible. It does not yet detect the broken state on its own: a daemon whose interpreter loses the package still dies at the next restart, and the user learns about it from clawmetry status or the dashboard banner. Only the macOS paths changed; systemd and the Windows scheduled task are untouched.

No-PRD: production bug fix for a founder-reported dead daemon; FLYWHEEL 0c scopes requirement-first to "any change bigger than a bug fix", and writing a record after implementing is the 2026-08-25 anti-pattern that section exists to prevent.

🤖 Generated with Claude Code

The sync daemon's LaunchAgent pins an absolute interpreter path at install
time. When that interpreter loses the clawmetry package the running daemon
survives (Python holds imported modules in memory) and dies at the next
restart; every relaunch then fails with ModuleNotFoundError.

The documented repair, `clawmetry sync --restart`, could not fix that:
_register_launchd wrote the corrected plist but never booted the job out
first. `launchctl bootstrap` refuses an already-registered label, and the
legacy `load -w` fallback also fails while exiting 0, so no return code
could detect it. launchd kept the definition it already held in memory and
we printed "Running in the background. Your data is syncing to the cloud."
over a crash-looping daemon.

Reproduced against a live registered job: bootstrap rc=5, load -w prints
"Load failed" and exits 0. Founder node 2026-10-14, daemon dead ~5h while
the dashboard showed "Step: starting" and every daemon-backed tab rendered
an empty state.

- bootout before bootstrap, the only sequence that re-reads the plist and
  therefore the only one that can re-pin a stale interpreter
- retry the bootstrap briefly, since bootout is asynchronous
- confirm the legacy path with a real lookup, not load's lying exit code
- report failure instead of a success line; _cmd_sync exits non-zero
- the dashboard registrar used `kickstart -k`, which restarts the stale
  definition and can never re-pin; it now shares the same helper

The NemoClaw sandbox registrar already did this correctly and is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@8090-software-factory

Copy link
Copy Markdown

⚠️ Drift Bot (ClawMetry): 2 potential drift finding(s)

1. Blueprint: Installer Scripts

File: clawmetry/cli.py:1894

The blueprint does not document the new launchd re-registration logic (bootout → bootstrap with retries and verification via launchctl print) that is now implemented in _register_launchd(). The blueprint should be updated to reflect that daemon registration now includes re-pinning the interpreter path when registration is re-run.

2. Blueprint: Local Observability Service

File: clawmetry/cli.py:1722

The blueprint documents daemon registration but does not specify that daemon registration failure (when launchd refuses the job) must result in a non-zero exit code. The updated _start_daemon() now returns bool and _cmd_sync() exits with code 1 on failure, but the blueprint contains no requirement for failure reporting or exit code behavior.

Comment thread clawmetry/cli.py

def _register_launchd(config: dict) -> None:
def _launchd_reload(plist_path, label: str) -> bool:
"""Load a (re)written LaunchAgent plist and report whether it took.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Drift Bot (ClawMetry) — Blueprint: Installer Scripts

The blueprint does not document the new launchd re-registration logic (bootout → bootstrap with retries and verification via launchctl print) that is now implemented in _register_launchd(). The blueprint should be updated to reflect that daemon registration now includes re-pinning the interpreter path when registration is re-run.

Comment thread clawmetry/cli.py
"""

system = __import__("platform").system()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Drift Bot (ClawMetry) — Blueprint: Local Observability Service

The blueprint documents daemon registration but does not specify that daemon registration failure (when launchd refuses the job) must result in a non-zero exit code. The updated _start_daemon() now returns bool and _cmd_sync() exits with code 1 on failure, but the blueprint contains no requirement for failure reporting or exit code behavior.

The reload unloads the old job before loading the rewritten one, which is
what makes the re-pin possible. It also means a registration that fails
every retry leaves nothing registered at that moment -- the previous code
could not make things worse, because it never unloaded.

Tell the operator that instead of leaving it to be inferred from a generic
"could not start" line, since the practical consequence is that collection
has stopped until they try again.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@8090-software-factory

Copy link
Copy Markdown

✅ Drift Bot (ClawMetry): no drift detected

Drift Bot analyzed the changed files against this project's blueprints and requirements and found no drift.

@vivekchand
vivekchand merged commit 2fbb7e9 into main Oct 14, 2026
56 of 57 checks passed
vivekchand added a commit that referenced this pull request Oct 14, 2026
#6402)

When the collector reports no progress for 90s the banner said, on macOS,
to run `launchctl kickstart -k`. That restarts the job definition launchd
already holds in memory. It never re-reads the plist, so it cannot repair
the most common reason the collector goes quiet: a LaunchAgent pinned to an
interpreter that can no longer import clawmetry.

On the node this was found on it produced 12 consecutive failed launches
and could never have produced anything else. The banner was handing the
user the one command guaranteed not to help, while the daemon stayed dead.

It now points at `clawmetry sync --restart`, which re-registers the job
(bootout then bootstrap, #6401) and so actually re-pins it, and then at
`clawmetry status`, which reported this condition correctly all along.
Both work on every OS; the old copy offered nothing to anyone not on macOS.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
vivekchand added a commit that referenced this pull request Oct 14, 2026
#6401, #6402) (#6403)

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant