Skip to content

Module crash should disconnect all clients #2630

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kim opened this issue Apr 17, 2025 · 1 comment · May be fixed by #2652
Open

Module crash should disconnect all clients #2630

kim opened this issue Apr 17, 2025 · 1 comment · May be fixed by #2652
Assignees
Labels
bug Something isn't working

Comments

@kim
Copy link
Contributor

kim commented Apr 17, 2025

A module can crash for a number of reasons, including:

  • persistence becomes fatally unwritable
  • the current replica is demoted as the leader
  • some unknown / unhandled panic

Desired behavior

When a module crashes, all currently connected clients should be disconnected (or, reset in some other way).

Observed behavior

All crash scenarios described above manifest in a panic that is intercepted by the ModuleHost executing a reducer call. The module is removed from the HostController and the unwinding is resumed.

This disconnects the caller, but not other clients connected to the same replica.

When those other clients attempt to call a reducer, a NoSuchModule error will be propagated to the client, but the client connection is not terminated (cf. subscribe.rs#L369). Note that module hot-swapping should never manifest in a NoSuchModule error at this point.

@kim
Copy link
Contributor Author

kim commented Apr 17, 2025

Downcasting the error might be a quick fix, but I'm more interested in a more general / robust solution.

@joshua-spacetime joshua-spacetime added the bug Something isn't working label Apr 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants