Fix flaky network observer test and harden notification delivery#2559
Conversation
testNetworkObserverNotifiesOnlyOnce used a 1-second expectation timeout, the only positive wait in the file not using the 10-second convention. Its callback path hops through a utility-QoS queue before reaching main, so a multi-second scheduler stall on a loaded CI VM (evidenced by a nominal 0.15s neighbor test taking 2.167s in the same failing run) blows the budget. Align it with the rest of the file; the once-only contract is still enforced by assertForOverFulfill. Also capture self strongly in NetworkObserverImpl.notify/cancel blocks. Delivery of an already-scheduled one-shot notification previously depended on the caller happening to retain the observer until the async block ran; a dropped reference silently swallowed the callback. The blocks are one-shot so the temporary strong capture cannot cycle, and the timeout work item stays weak so a pending timer does not pin an abandoned observer. TestNetworkObserver mirrors the same change.
The PR run reproduced the same failure class on a different test (testDidFinishDownloadingDelegateCalledOncePerErroredDownload, a local Nocilla-stubbed test with a 3s budget): its suite shows 11.2s of CPU time against 31.4s wall, i.e. ~20s of VM-level stalling. No per-test timeout survives that, so inflating individual budgets is whack-a-mole. Pass number_of_retries to scan (-retry-tests-on-failure) instead; green runs are unaffected and a genuine regression still fails every attempt.
|
The first CI round on this PR delivered a decisive extra data point: The numbers from that job make the mechanism unambiguous: that suite executed 36 tests in 11.2s CPU / 31.4s wall (~20s of pure stall), and the whole bundle ran 387 tests in 55.3s CPU / 96.6s wall. The 26.2-iOS leg's runner stalls at hypervisor level for tens of seconds - no reasonable per-test timeout survives that, so bumping budgets one by one is whack-a-mole. Added Why this leg specifically: plausibly the macos-26 image ships with the newest simulator runtime warmed up, while the iOS 26.2 runtime needs first-boot dyld cache work that drags the whole VM - which also matches "rerun always passes" (second boot is warm). |
Round-2 CI on the 26.2 leg turned three retries of testNetworkRetryStrategyWaitsForReconnection into 10s timeouts while neighboring tests passed in milliseconds, then delivered the callbacks up to 36s late - misattributed into subsequent tests (XCTAssertNotNil at line 407 reported inside testNetworkObserverDoesNotNotifyAfterCancel). That is not a whole-VM stall: it is the per-observer utility-QoS serial queue being starved for tens of seconds while default+ bands stay live. The one-shot transition (flip isFinished, cancel the timer, deregister) is a microsecond critical section; running it as an async hop on a starvable queue opened an unbounded late-delivery window. Decide it synchronously under NSLock and keep only the user-facing callback on DispatchQueue.main. The utility queue leaves the delivery path entirely; cancel-then-notify suppression now takes effect immediately instead of relying on queue ordering. The timeout timer moves to the global utility queue and funnels through the same atomic finish(). TestNetworkObserver mirrors the change. Monitor-side dispatch is untouched: NetworkMonitor.removeObserver stays an async barrier (safe from any thread) and TestNetworkMonitor's main-queue hop keeps notify out of its sync-barrier critical section.
|
Round 2 falsified the pure "runner stall" theory and pinpointed the real defect. Timeline from the failing job: Neighboring tests passed in milliseconds throughout, so this is not a whole-VM stall: the per-observer The defect is structural: the one-shot transition (flip Local verification (iPhone 17 / iOS 26.2): 30 iterations x full |
|
Verification status after the lock-based fix (
Plus locally on iPhone 17 / iOS 26.2: 30x full The late-delivery failure mode is now structurally impossible (no starvable queue left in the delivery path); unrelated stall-induced timeouts in other suites are absorbed by |
Problem
RetryStrategyTests.testNetworkObserverNotifiesOnlyOncefails intermittently on CI with:Example: run 29687516319 (Xcode 26.2, iOS 26.2). A rerun reliably passes.
Root cause
A CI-side scheduler stall, not an implementation bug. In the same failing run, the neighboring
testNetworkRetryStrategyWaitsForReconnection(nominally ~0.15s: one 0.1sasyncAfterplus a few queue hops) took 2.167 seconds - the runner VM stalled for ~2s right around the failure. The failing test's callback path hops through a.utilityQoS serial queue before reaching main (~5ms when healthy), and its 1-second expectation budget is the only positive wait in the file that doesn't follow thetimeout: 10convention; it was seemingly copied from the inverted sibling test above it, where a short window is intentional.Historical failures of the same test family (
testNetworkRetryStrategyWaitsForReconnection/testNetworkRetryStrategyCancelsPreviousObserver, June, Xcode 26.5/26.0.1) all predate #2536-era fixes (7730ef18,b7ce92aa); after those landed, this 1s timeout is the only remaining flake, and the Xcode 26.2 correlation is a single-sample coincidence.Changes
testNetworkObserverNotifiesOnlyOncewait from 1s to 10s, matching every other positive expectation in the file. The once-only contract is enforced byassertForOverFulfill, which is independent of the timeout value;waitForExpectationsreturns immediately on fulfillment, so the happy path stays milliseconds.NetworkObserverImpl.notify()/cancel()now captureselfstrongly in their async blocks. Previously, delivery of an already-scheduled one-shot notification depended on the caller happening to retain the observer until the block executed; dropping the last reference in that window silently swallowed the callback. The blocks are one-shot so the temporary strong capture cannot form a lasting cycle, and the timeout work item keeps its weak capture so a pending 30s timer never pins an abandoned observer.TestNetworkObservermirrors the change.Verification
On iPhone 17 / iOS 26.2 simulator, with all CPU cores saturated to try to reproduce the starvation:
KingfisherTestssuite: 388/388 passedswift build(macOS/SPM): clean