Repository navigation
Stabilize scheduled fault campaigns - #27
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The scheduled fault campaign has failed on every run since it was enabled. Two protocol mutants consistently survive because they are assigned to nearby regressions that do not exercise the faulty behavior. The repeated distributed suite also observes three valid lifecycle windows as failures: recovery can begin before a killed owner commits its leave, exact authority can overtake an incremental cluster-close message, and the test TCP route table is briefly absent while its manager restarts.
These failures keep the qualification job red and prevent the downstream Jepsen matrix from running, despite not demonstrating violations of the intended protocol invariants.
Fix
Associate the generation-hint and journal-repair mutants with the existing regressions that directly exercise those faults. Make reconnect recovery wait for the authoritative owner removal, and verify lane-restart repair from the durable state that matters rather than one possible queued control-message shape.
Treat a missing test TCP route table during supervised manager replacement as disconnected. This matches the adapter's existing outgoing-send behavior and lets eventual reconnection polling span the table replacement window.
Supporting information
This changes qualification and test synchronization only; Group's production protocol behavior is unchanged. The affected scheduled run is https://github.com/phoenixframework/group/actions/runs/34949094333.