Skip to content

feat(core): resilient daemon restart with exponential backoff reconnection #33432

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

feat(core): resilient daemon restart with exponential backoff reconnection #33432

wants to merge 1 commit into from

Conversation

@FrozenPandaz
Copy link
Collaborator

@FrozenPandaz FrozenPandaz commented Nov 10, 2025

Current Behavior

The daemon currently depends on client requests failing with a LOCK_FILES_CHANGED error to trigger a restart. This creates several issues:

  • The daemon may stay running with stale dependencies if no requests come in
  • The client must wait for a request to fail to trigger reconnection
  • Special-case error handling is scattered across the client code

Expected Behavior

The daemon should proactively restart itself when lock files change, and the client should gracefully reconnect with exponential backoff for any server shutdown scenario.

Changes

Client-Side: Exponential Backoff Reconnection

  • Add handleConnectionError() method that retries with exponential backoff (10ms → 5000ms, 30 attempts max)
  • Preserve pending messages during reconnection and resend them once the new daemon is available
  • Remove special-case handling for LOCK_FILES_CHANGED and NX_VERSION_CHANGED errors
  • Remove retryMessageAfterNewDaemonStarts() method as it's no longer needed

Server-Side: Self-Restart on Lock File Changes

  • Add startNewDaemonInBackground() to spawn a replacement daemon before shutdown
  • Add handleServerProcessTerminationWithRestart() for restartable shutdown scenarios
  • Detect lock file changes and proactively start a new daemon before responding with an error
  • Keep version change handling simple (just exit, no restart)

Benefits

  • More Resilient: Client recovers from any server shutdown, not just specific errors
  • Cleaner Architecture: Server manages its own lifecycle, client doesn't need special cases
  • No Request Dependency: Daemon doesn't wait for requests to detect changes
  • Reduced Error Spam: Exponential backoff prevents connection error floods
  • Future-Proof: Foundation for other restart scenarios (plugins, config changes)

Related Issue(s)

Part of daemon rework to make restarts more reliable and resilient.

@vercel
Copy link

vercel bot commented Nov 10, 2025 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Preview Updated (UTC)
nx-dev Ready Ready Preview Nov 21, 2025 3:56pm

@netlify
Copy link

netlify bot commented Nov 10, 2025 •
edited
Loading

✅ Deploy Preview for nx-docs ready!

Name Link
🔨 Latest commit 9d8d5e7
🔍 Latest deploy log https://app.netlify.com/projects/nx-docs/deploys/69208aa1cf9fca0008ac0b2c
😎 Deploy Preview https://deploy-preview-33432--nx-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@nx-cloud
Copy link
Contributor

nx-cloud bot commented Nov 11, 2025 •
edited
Loading

View your CI Pipeline Execution ↗ for commit 9d8d5e7

Command Status Duration Result
nx affected --targets=lint,test,test-kt,build,e... ❌ Failed 35m 38s View ↗
nx run-many -t check-imports check-lock-files c... ✅ Succeeded 2m 35s View ↗
nx-cloud record -- nx-cloud conformance:check ✅ Succeeded 12s View ↗
nx-cloud record -- nx format:check ✅ Succeeded 3s View ↗
nx-cloud record -- nx sync:check ✅ Succeeded <1s View ↗

☁️ Nx Cloud last updated this comment at 2025-11-21 16:31:28 UTC

@FrozenPandaz FrozenPandaz force-pushed the daemon-rework branch 3 times, most recently from ace03dc to e7ef248 Compare November 20, 2025 17:26
@FrozenPandaz FrozenPandaz force-pushed the daemon-rework branch 2 times, most recently from 9b6e5ec to d2bf2f0 Compare November 20, 2025 18:18
@FrozenPandaz
fix(core): improve daemon reliability with reconnection and socket ha…
9d8d5e7 
…ndling

- Add exponential backoff reconnection for daemon client
- Implement version check handshake in socket messenger
- Improve socket error handling and prevent hanging on errors
- Keep watch process alive during file change listening
- Simplify socket messenger by removing socketPath parameter
- Move socket creation into listen() to prevent unhandled errors
- Add graceful daemon reconnection with user feedback
- Restart daemon on lock file and version changes
- Prevent watch connections from exiting during reconnection
- Improve in-flight request handling during daemon reconnection
- Add periodic check for daemon being outdated
- Extract shared connections for file watchers and graph listeners
- Simplify pending message handling in reconnection
- Silence noisy reconnection logs during normal restarts
- Fix file-server executor to use output.log instead of logger
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@FrozenPandaz