fix(discord): tolerate poisoned delivery-state mutex - #352
Merged
Merged
Conversation
`src/discord.rs` was the only production path locking its shared state with `expect(...)`. Since that state guards the rate limiter, the per-target circuit breakers, and the DLQ buffer, a single panic unwinding inside any critical section poisoned the mutex and made every later `allow_request`, `rate_limit_delay`, `record_success`, `record_failure`, and DLQ bury panic for the remaining lifetime of the daemon -- permanently destroying the whole Discord delivery lane, including the DLQ capture meant to preserve undelivered messages. Route all five sites through a single poison-tolerant `state()` accessor (`PoisonError::into_inner`), matching the convention already used by the daemon, dispatch, lane, subscription, git, tmux, and lifecycle paths. All three guarded structures are individually recoverable, so recovery degrades to possibly-stale counters instead of an unrecoverable panic loop. Adds a regression test that poisons the state from a panicking thread and asserts the limiter, circuit-breaker open transition, and DLQ paths keep working. The test panics with `PoisonError` against the previous code. Fixes #351
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #351.
What
src/discord.rswas the only production path in the crate locking its sharedMutexwithexpect(...)(5 sites:allow_request,rate_limit_delay,record_success,record_failure, DLQ bury). All five now go through one poison-tolerantstate()accessor usingPoisonError::into_inner.Why
DiscordStateguards the rate limiter, the per-target circuit breakers, and the DLQ buffer. One panic unwinding while that guard was held poisoned the mutex, so every later Discord delivery decision and DLQ bury panicked for the rest of the daemon's lifetime — permanently losing the whole Discord lane, including the DLQ capture that exists precisely to preserve undelivered messages.The rest of the crate already tolerates poisoning (
daemon.rs,dispatch.rs,gjc_lane.rs,source/subscription.rs,source/git.rs,source/tmux.rs,lifecycle.rs), so this aligns Discord with the existing convention rather than introducing a new policy. All three guarded structures are individually recoverable, so recovery degrades to possibly-stale counters instead of an unrecoverable panic loop.Verification
cargo test --bin clawhip→ 1101 passed, 0 failedcargo fmt --check→ cleancargo clippy --all-targets -- -D warnings→ cleandiscord::tests::poisoned_delivery_state_still_serves_limiter_circuit_and_dlqpoisons the state from a panicking thread, then asserts the limiter delay, the circuit-breaker open transition, and the DLQ path all still work.expect(...), that test fails withpanicked at src/discord.rs:751: discord state lock: PoisonError { .. }, confirming the test actually covers the regression.Base
dev@c4774562c6b073d4d6e1481aeb10ee8aa68afbad. No config/schema/behavioral change on the healthy path.—
[repo owner's gaebal-gajae (clawdbot) 🦞]