Skip to content

Commit 382caaa

Browse files
beastoinclaude
andauthored
Fix WebSocket transcription disconnects — 64K Sentry events (#6193) (#6220)
* fix(desktop): robust WebSocket reconnection in TranscriptionService Fixes #6193 — 64K Sentry events from WebSocket transcription disconnects. Root causes fixed: - Race condition: replaced 0.5s hardcoded delay with URLSessionWebSocketDelegate handshake detection (didOpenWithProtocol) + 10s connect timeout - Audio loss: added ring buffer (960KB/30s TTL) to hold audio during reconnect, replayed on successful reconnection - Permanent failure: removed 10-attempt reconnect cap, now retries indefinitely with exponential backoff + jitter (max 60s) while recording is active - Thread safety: all mutable connection state behind serial DispatchQueue, ConnectionState enum replaces bare Bool - Stale callbacks: generation token discards delegate callbacks from old connections Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(desktop): graceful WebSocket close forwarding in proxy Part of #6193 — when one side of the Deepgram WS proxy disconnects, forward a close frame to the other side with a 5s timeout instead of abruptly dropping both connections. Prevents "Connection reset by peer" errors on the Swift client. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(desktop): changelog entry for WebSocket reconnect fix Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(desktop): address review — gate auth on generation, idempotent disconnect Review cycle fixes for #6201: - Gate proxy auth Task and connectWithAuth on generation + shouldReconnect to prevent zombie connections after stop() - Make handleDisconnection idempotent: only transitions from .connected or .connecting states, preventing duplicate onDisconnected notifications and inflated reconnect counts from concurrent failure callbacks - Validate generation in didOpenWithProtocol to reject stale handshakes Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(desktop): bump generation on teardown, salvage partial audio buffer Review cycle 2 fixes for #6201: - Bump _connectionGeneration in both disconnect() and handleDisconnection() so in-flight receiveMessage/keepalive callbacks are invalidated, preventing stale transcript delivery after stop() or during reconnect gap - Salvage partial audioBuffer contents into reconnectBuffer on disconnect, preventing the last ~100ms audio chunk from being lost or replayed out of order after reconnection Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(desktop): re-buffer unsent chunks on replay failure Review cycle 3 fix for #6201: - On replay send error, re-buffer the failed chunk and all remaining chunks back into reconnectBuffer, then trigger handleDisconnection() to reconnect and retry. Previously, drained chunks were permanently lost if the socket failed during replay. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(desktop): expose ring buffer and backoff for testability Extract reconnectDelay() as static method and make ReconnectAudioRingBuffer internal for @testable import. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test(desktop): add unit tests for ring buffer and backoff calculation 13 tests covering: - ReconnectAudioRingBuffer: append/drain, TTL eviction, byte-cap eviction, oversize chunk truncation, prune, empty data handling - reconnectDelay(): exponential growth, max backoff cap, jitter bounds, attempt zero edge case All 13 tests pass. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(desktop): update OnboardingFlowTests for new migratedStep params Add missing hasRemovedNotificationStep, hasInsertedFloatingBarShortcutStep, and hasMigratedPagedIntro parameters to fix pre-existing compile error. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(desktop): update OnboardingFlowTests for current 17-step flow Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(desktop): unwind state on invalid URL, sequential replay to prevent duplicates - Invalid URL guards in connectWithAuth now call handleDisconnection() instead of bare return, preventing permanent .connecting wedge state - Replay sends chunks sequentially (callback-chained) so only the first failure re-buffers remaining chunks, preventing duplicate audio from concurrent failures Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(desktop): add test accessors for state machine verification Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test(desktop): add state machine, idempotency, and URL construction tests - 7 TranscriptionServiceStateTests: initial state, stop transitions, handleDisconnection idempotency from all 4 states - 3 URLConstructionTests: empty base, malformed base, valid base Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(desktop): add hasReorderedTrustStep param to OnboardingFlowTests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(desktop): prevent replay interleaving and cap backoff at 32s Add _isReplaying flag to gate live sendAudio() calls during buffered chunk replay — prevents interleaving that could corrupt transcript order. Cap jitter range to 0.8...1.0 and clamp final delay to maxBackoff (32s) so reconnect never exceeds documented maximum. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(desktop): correct OnboardingFlowTests step order to match main Update expected step order to Name, Language, Trust (matching current OnboardingFlow.steps after trust step reorder on main). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(desktop): drain chunks accumulated during replay to prevent stranding After replayChunksSequentially finishes the initial batch, check if sendAudio() appended new data to reconnectBuffer while _isReplaying was true. If so, drain and continue replaying before clearing the flag. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(desktop): add test accessors for replay gating and reconnect buffer Add testIsReplaying, testSetIsReplaying, testAppendToReconnectBuffer, and testDrainReconnectBuffer accessors for @testable import. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test(desktop): add replay gating and disconnect buffer salvage tests Test that sendAudio buffers data in reconnectBuffer during replay, does not buffer when not replaying, _isReplaying flag initializes correctly, and reconnect buffer survives handleDisconnection. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test(desktop): add ProxyCloseOrigin enum variant test Verify all four ProxyCloseOrigin variants exist with distinct Debug output, covering the new close-origin tracking in proxy_ws_bidirectional. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Simplify WS reconnect fix: remove audio buffering, keep connection state management Remove ReconnectAudioRingBuffer, replay logic, and _isReplaying gating. Audio is now silently dropped during disconnects (buffering is a future phase). Keep: thread-safe ConnectionState, URLSessionWebSocketDelegate handshake, infinite reconnect with backoff+jitter, idempotent handleDisconnection, generation tokens for stale callback discard. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove ring buffer and replay tests, add sendAudio drop tests Remove ReconnectAudioRingBufferTests, ReplayGatingTests, and DisconnectBufferSalvageTests. Add SendAudioDropTests verifying audio is silently dropped in disconnected/reconnecting/connecting states. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update changelog to remove audio buffering mention Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent a29220e commit 382caaa

4 files changed

Lines changed: 561 additions & 116 deletions

File tree

desktop/Backend-Rust/src/routes/proxy.rs

Lines changed: 86 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -253,7 +253,17 @@ async fn deepgram_ws_proxy(
253253
}))
254254
}
255255

256+
/// Which side of the proxy terminated first
257+
#[derive(Debug)]
258+
enum ProxyCloseOrigin {
259+
ClientClosed,
260+
UpstreamClosed,
261+
ClientError,
262+
UpstreamError,
263+
}
264+
256265
/// Bidirectional WebSocket proxy between client (axum) and upstream (tokio-tungstenite).
266+
/// When one side closes or errors, a close frame is forwarded to the other side before teardown.
257267
async fn proxy_ws_bidirectional(
258268
client_socket: axum::extract::ws::WebSocket,
259269
upstream_url: &str,
@@ -278,47 +288,71 @@ async fn proxy_ws_bidirectional(
278288

279289
// Client → Upstream
280290
let client_to_upstream = async {
281-
while let Some(Ok(msg)) = client_stream.next().await {
282-
let tung_msg = match msg {
283-
AxumMsg::Text(t) => TungMsg::Text(t),
284-
AxumMsg::Binary(b) => TungMsg::Binary(b),
285-
AxumMsg::Ping(p) => TungMsg::Ping(p),
286-
AxumMsg::Pong(p) => TungMsg::Pong(p),
287-
AxumMsg::Close(_) => {
288-
let _ = upstream_sink.close().await;
289-
return;
291+
while let Some(result) = client_stream.next().await {
292+
match result {
293+
Ok(msg) => {
294+
let tung_msg = match msg {
295+
AxumMsg::Text(t) => TungMsg::Text(t),
296+
AxumMsg::Binary(b) => TungMsg::Binary(b),
297+
AxumMsg::Ping(p) => TungMsg::Ping(p),
298+
AxumMsg::Pong(p) => TungMsg::Pong(p),
299+
AxumMsg::Close(_) => {
300+
let _ = upstream_sink.close().await;
301+
return ProxyCloseOrigin::ClientClosed;
302+
}
303+
};
304+
if upstream_sink.send(tung_msg).await.is_err() {
305+
return ProxyCloseOrigin::UpstreamError;
306+
}
290307
}
291-
};
292-
if upstream_sink.send(tung_msg).await.is_err() {
293-
return;
308+
Err(_) => return ProxyCloseOrigin::ClientError,
294309
}
295310
}
311+
ProxyCloseOrigin::ClientClosed
296312
};
297313

298314
// Upstream → Client
299315
let upstream_to_client = async {
300-
while let Some(Ok(msg)) = upstream_stream.next().await {
301-
let axum_msg = match msg {
302-
TungMsg::Text(t) => AxumMsg::Text(t),
303-
TungMsg::Binary(b) => AxumMsg::Binary(b),
304-
TungMsg::Ping(p) => AxumMsg::Ping(p),
305-
TungMsg::Pong(p) => AxumMsg::Pong(p),
306-
TungMsg::Close(_) => {
307-
let _ = client_sink.close().await;
308-
return;
316+
while let Some(result) = upstream_stream.next().await {
317+
match result {
318+
Ok(msg) => {
319+
let axum_msg = match msg {
320+
TungMsg::Text(t) => AxumMsg::Text(t),
321+
TungMsg::Binary(b) => AxumMsg::Binary(b),
322+
TungMsg::Ping(p) => AxumMsg::Ping(p),
323+
TungMsg::Pong(p) => AxumMsg::Pong(p),
324+
TungMsg::Close(_) => {
325+
let _ = client_sink.close().await;
326+
return ProxyCloseOrigin::UpstreamClosed;
327+
}
328+
TungMsg::Frame(_) => continue,
329+
};
330+
if client_sink.send(axum_msg).await.is_err() {
331+
return ProxyCloseOrigin::ClientError;
332+
}
309333
}
310-
TungMsg::Frame(_) => continue,
311-
};
312-
if client_sink.send(axum_msg).await.is_err() {
313-
return;
334+
Err(_) => return ProxyCloseOrigin::UpstreamError,
314335
}
315336
}
337+
ProxyCloseOrigin::UpstreamClosed
316338
};
317339

318-
// Run both directions concurrently; when either ends, drop both
319-
tokio::select! {
320-
_ = client_to_upstream => {},
321-
_ = upstream_to_client => {},
340+
// Run both directions concurrently; when either ends, gracefully close the other side
341+
let origin = tokio::select! {
342+
origin = client_to_upstream => origin,
343+
origin = upstream_to_client => origin,
344+
};
345+
346+
// Forward close frame to the surviving side with a timeout to prevent hanging
347+
let close_timeout = std::time::Duration::from_secs(5);
348+
tracing::debug!("deepgram_ws_proxy: proxy ended ({:?})", origin);
349+
match origin {
350+
ProxyCloseOrigin::UpstreamClosed | ProxyCloseOrigin::UpstreamError => {
351+
let _ = tokio::time::timeout(close_timeout, client_sink.close()).await;
352+
}
353+
ProxyCloseOrigin::ClientClosed | ProxyCloseOrigin::ClientError => {
354+
let _ = tokio::time::timeout(close_timeout, upstream_sink.close()).await;
355+
}
322356
}
323357

324358
Ok(())
@@ -555,4 +589,27 @@ mod tests {
555589
let msg = parsed["error"]["message"].as_str().unwrap().to_lowercase();
556590
assert!(msg.contains("resource exhausted"));
557591
}
592+
593+
// --- ProxyCloseOrigin ---
594+
595+
#[test]
596+
fn proxy_close_origin_debug_variants() {
597+
// Verify all variants exist and produce distinct debug output
598+
let variants = [
599+
ProxyCloseOrigin::ClientClosed,
600+
ProxyCloseOrigin::UpstreamClosed,
601+
ProxyCloseOrigin::ClientError,
602+
ProxyCloseOrigin::UpstreamError,
603+
];
604+
let debug_strs: Vec<String> = variants.iter().map(|v| format!("{:?}", v)).collect();
605+
assert_eq!(debug_strs.len(), 4);
606+
// All distinct
607+
let unique: std::collections::HashSet<&String> = debug_strs.iter().collect();
608+
assert_eq!(unique.len(), 4, "All ProxyCloseOrigin variants should have distinct Debug output");
609+
// Verify expected names
610+
assert!(debug_strs.contains(&"ClientClosed".to_string()));
611+
assert!(debug_strs.contains(&"UpstreamClosed".to_string()));
612+
assert!(debug_strs.contains(&"ClientError".to_string()));
613+
assert!(debug_strs.contains(&"UpstreamError".to_string()));
614+
}
558615
}

desktop/CHANGELOG.json

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
{
2-
"unreleased": [],
2+
"unreleased": [
3+
"Fixed WebSocket transcription disconnects: proper handshake detection, unlimited retry with backoff, and thread-safe connection state"
4+
],
35
"releases": [
46
{
57
"version": "0.11.218",
@@ -103,7 +105,7 @@
103105
"version": "0.11.202",
104106
"date": "2026-03-31",
105107
"changes": [
106-
"Fixed WebSocket transcription disconnects: proper handshake detection, audio buffering during reconnection, unlimited retry with backoff, and thread-safe connection state",
108+
"Fixed WebSocket transcription disconnects: proper handshake detection, unlimited retry with backoff, and thread-safe connection state",
107109
"Fixed UI freezes caused by dock tile updates when receiving support messages"
108110
]
109111
},

0 commit comments

Comments
 (0)