You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Smoke Tests in the iOS workflow is not deterministic on main. Two alert-observation XCTests fail on the runner host without any Apple source changing, so every PR that reaches that step has to be re-run before a reviewer can trust it.
Observed on main
main@973b74cc14 (push run 34741577232, job 103682270067, step Run targeted iOS runner XCTest regressions), RunnerTests+AlertObservationTests.swift:72:
XCTAssertEqual failed: ("false") is not equal to ("true")
- Optional(… code: "ALERT_DEADLINE_EXCEEDED", message: "alert dismiss exhausted its deadline")
XCTAssertEqual failed: ("First actions: 0; replacement actions: 0") is not equal to ("First actions: 1; replacement actions: 0")
Failing test: testAlertCannotProveAnIdenticalReplacementAndDoesNotActivateIt. Attempt 2 of the identical run and commit passed at 06:46 UTC, so the second case is a flake rather than a regression.
Both symptoms share one shape: a presented alert reports no actions, and the assertion times out through ALERT_DEADLINE_EXCEEDED.
Separate symptom on the same step family, worth tracking apart from the two above: main@8e8eeb2ced (run 34712516058) failed Run fixture-backed iOS simulator E2E smoke with an assertion raised at test/integration/ios-simulator-e2e/live-automation-scenario.ts:33 (observeFixtureHome).
Required behavior
The alert-observation assertions in apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+AlertObservationTests.swift pass or fail on their own terms, not on simulator alert presentation timing.
A presented system alert that the test can see must expose its button list to the probe, or the probe must report the empty-button case as a distinct typed outcome instead of consuming the whole deadline.
Whatever the resolution, an unchanged main must not require a manual re-run to go green.
Observable completion conditions
A diagnosis recorded here naming the mechanism (alert not yet realized when the action list is read, deadline sized below host latency, or a genuine runner-side ordering bug).
If it is timing: the probe or deadline change lands and the iOS workflow's Smoke Tests job is green across at least five consecutive main pushes, with no manual re-runs.
If it is a real defect: a failing-then-passing XCTest that pins it, plus the same five-run streak.
The observeFixtureHome case is either fixed or split into its own issue with the above evidence linked.
Dependencies
Reproduce on a hosted macOS runner or a local simulator; the failure does not appear in unit or check:affected runs, so this is device-facing and docs/agents/device-verification.md applies.
Related history for the alert horizon: ADR-tracked alert deadline work behind testAlertResolutionCannotBypassRequestedDeadline and testAlertDeadlineBeforeActivationLeavesTheOriginalUntouched, which pass in the same runs where the two above fail.
Discovered while babysitting #2533, whose diff touched no Apple source.
Purpose
Smoke Testsin theiOSworkflow is not deterministic onmain. Two alert-observation XCTests fail on the runner host without any Apple source changing, so every PR that reaches that step has to be re-run before a reviewer can trust it.Observed on
mainmain@973b74cc14(push run 34741577232, job 103682270067, stepRun targeted iOS runner XCTest regressions),RunnerTests+AlertObservationTests.swift:72:Failing test:
testAlertDismissDoesNotActivateAReplacementWithTheSameTitle.PR docs: fix phantom specifiers, the duplicate ADR 0019, and the Node floor #2533 head
a16821b661(run 34742468875 attempt 1, docs-only diff), same file at:75and:80:Failing test:
testAlertCannotProveAnIdenticalReplacementAndDoesNotActivateIt. Attempt 2 of the identical run and commit passed at 06:46 UTC, so the second case is a flake rather than a regression.Both symptoms share one shape: a presented alert reports no actions, and the assertion times out through
ALERT_DEADLINE_EXCEEDED.Separate symptom on the same step family, worth tracking apart from the two above:
main@8e8eeb2ced(run 34712516058) failedRun fixture-backed iOS simulator E2E smokewith an assertion raised attest/integration/ios-simulator-e2e/live-automation-scenario.ts:33(observeFixtureHome).Required behavior
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+AlertObservationTests.swiftpass or fail on their own terms, not on simulator alert presentation timing.mainmust not require a manual re-run to go green.Observable completion conditions
iOSworkflow'sSmoke Testsjob is green across at least five consecutivemainpushes, with no manual re-runs.observeFixtureHomecase is either fixed or split into its own issue with the above evidence linked.Dependencies
check:affectedruns, so this is device-facing anddocs/agents/device-verification.mdapplies.testAlertResolutionCannotBypassRequestedDeadlineandtestAlertDeadlineBeforeActivationLeavesTheOriginalUntouched, which pass in the same runs where the two above fail.Discovered while babysitting #2533, whose diff touched no Apple source.