From 4296302712f2eb543c773dd028bf98fab369a6b9 Mon Sep 17 00:00:00 2001 From: onevcat Date: Tue, 28 Jul 2026 15:17:41 +0900 Subject: [PATCH 1/2] fix: never pin a HID transport auto-selected inside the boot-attach window dtuhidd attaches 0-3 s after boot on a simulator booted while Device Hub is open, so upstream's transport auto-selection races it: a HID verb landing inside the window picks Indigo, the connection is cached under the boot token, and the wrong transport is silently pinned for the entire boot (Indigo reports success without delivering, so recovery never fires). Gate the cache instead of the construction: a connection whose transport was auto-selected while launchd_sim uptime is under 15 s is served to the command that created it but never reused, so the next command re-derives the selection after the attach window closes. This is the issue's skip-the-cache variant, implemented as cache-but-don't-reuse so every connection stays on the explicit disconnect path. A forced SIM_USE_HID_TRANSPORT selection cannot race and is cached unconditionally; an unknowable launchd_sim identity keeps the marker-fallback caching behavior. Also wire the diagnostics prerequisite: SIM_USE_DEBUG=1 turns on the default logger's stderr sink so the transport-selection signals are visible in normal runs and in the daemon logfile. Fixes #67 Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Rir4WwrPShf5DYzziJ8RyC Signed-off-by: onevcat --- CHANGELOG.md | 2 + README.md | 10 +++- .../iOSSimBackend/HID/HIDBootIdentity.swift | 23 +++++++++ Sources/iOSSimBackend/HID/HIDInteractor.swift | 37 +++++++++++--- .../iOSSimBackend/Types/SimUseLogger.swift | 9 +++- Tests/HIDBootIdentityTests.swift | 51 +++++++++++++++++++ 6 files changed, 122 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9e903962..e20206f2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - The agent-eval suite can now pin exactly which sim-use binary a run evaluates: `make eval ARGS="-b "` / `scripts/eval.sh --sim-use ` / `run.py --sim-use ` (default remains whatever `sim-use` resolves to on PATH). The wrapper and runner print `sim-use under test: ()` up front and the report header records it, so a run can never silently exercise the wrong binary — and development builds under `.build/` can be evaluated directly. New repo skill `.claude/skills/run-evals/` orchestrates the whole flow for agents and contributors: environment prep (Device Hub closed, fixtures installed), binary selection, cost confirmation, and verdict triage; `e2e/agent-evals/README.md` documents the new prereqs and flags. - `SIM_USE_HID_TRANSPORT=indigo|dtuhid` debug override to force a specific iOS HID transport (default: automatic per-boot selection). The per-UDID daemon keeps the environment it was spawned with — combine with `SIM_USE_NO_DAEMON=1` or restart the daemon for ad-hoc experiments. +- `SIM_USE_DEBUG=1` turns on the default logger's stderr sink (with debug-level detail), making internal info-lines — notably the HID transport-selection signals logged at connection creation — visible in normal runs and in the daemon logfile. Same daemon-environment caveat as `SIM_USE_HID_TRANSPORT`. (#67) - `describe-ui --no-raw` (top-level, `ios describe-ui`, and `android describe-ui`): with `--json`, omit the raw accessibility tree from the envelope. `data.raw` typically dominates the payload on real app screens and is only useful for debugging sim-use itself; `outline` / `entries` / `lists` are unaffected. - *Keeping output small* section in `skills/sim-use/SKILL.md`: steers agents to prefer the text outline, pair `--json` with `--no-raw`, verify via outline instead of screenshots, reuse the verify read as the next observe, and batch known sequences. - Viewer: `GET /api/snapshot` now forwards the CLI's calibrated interface orientation as `screen.orientation` (omitted when `describe-ui` reports none — Android, legacy daemons), and the SPA tags a rotated screen next to the W×H readout, e.g. `874×402 (landscape-right)`. (#57) @@ -26,6 +27,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- The iOS HID transport auto-selection no longer races dtuhidd's boot-time attach (#67). dtuhidd appears 0–3 s after boot on a simulator booted while Device Hub is open; a HID verb landing inside that window probed the process tree too early, auto-selected the legacy Indigo transport, and the cached connection pinned the wrong transport for the entire boot — `type` exited 0 while delivering nothing, with no error to trigger recovery. A connection whose transport was auto-selected while `launchd_sim` uptime is under 15 s is now served once but never reused, so the next command re-derives the selection after the attach window closes. Steady-state behavior (uptime ≥ 15 s at first HID verb) is unchanged. - Android `record-video` now preserves a static final frame through the requested stop time, and waits for stdout chunks already being delivered before finalizing the MP4. - `record-video` frame rate is no longer capped at the ~8–10 fps the screenshot-polling loop topped out at (iOS honors `--fps` up to 60 with smooth constant-rate playback; an Android emulator captured ~50 fps under motion). Recording no longer burns CPU decoding/re-encoding every frame. - Building the XCFrameworks now requires [XcodeGen](https://github.com/yonaskolb/XcodeGen) (`brew install xcodegen`) — the idb checkout generates `FBSimulatorControl.xcodeproj` from `project.yml`. The frameworks are built without library evolution, so `build_products/` is locked to the toolchain that produced it: re-run `./scripts/build.sh dev` after switching Xcode versions (CI keys its cache on the Xcode version for the same reason). diff --git a/README.md b/README.md index b5a4a630..4f57aebc 100644 --- a/README.md +++ b/README.md @@ -126,9 +126,15 @@ it), **including Device Hub workflows**: Hub was open) is driven through dtuhidd's CoreDevice HID service, and everything else through the legacy SimulatorKit path. No reboot dance, no guard errors — `tap` / `type` / `swipe` work in both states. +- A selection made within 15 s of boot (dtuhidd attaches 0–3 s after + boot, so a `simctl boot && sim-use type` script can probe too early) + is used once but not cached; the next command re-derives it, so an + early wrong pick never sticks for the boot. - `SIM_USE_HID_TRANSPORT=indigo|dtuhid` forces a specific transport for - debugging (combine with `SIM_USE_NO_DAEMON=1` — the per-UDID daemon - keeps the environment it was first spawned with). + debugging, and `SIM_USE_DEBUG=1` surfaces the transport-selection + signals (and other internal info-lines) on stderr / in the daemon + logfile (combine either with `SIM_USE_NO_DAEMON=1` — the per-UDID + daemon keeps the environment it was first spawned with). - Xcode 27 no longer bundles Simulator.app; the one from an Xcode 26.x install still works for viewing simulators, as does Device Hub itself. - Work record: `docs/ai/xxxx-xcode27-support/README.md`. diff --git a/Sources/iOSSimBackend/HID/HIDBootIdentity.swift b/Sources/iOSSimBackend/HID/HIDBootIdentity.swift index 41810d15..7a28874e 100644 --- a/Sources/iOSSimBackend/HID/HIDBootIdentity.swift +++ b/Sources/iOSSimBackend/HID/HIDBootIdentity.swift @@ -63,6 +63,29 @@ enum HIDBootIdentity { } } + /// The window after boot inside which upstream's HID transport + /// auto-selection cannot be trusted: dtuhidd attaches 0–3 s after + /// `launchd_sim` on a simulator booted while Device Hub is open + /// (issue #67), so a selection probed before it appears resolves + /// to Indigo on a dtuhidd-suppressed simulator — and Indigo + /// reports success without delivering, so no recovery path ever + /// rebuilds it. 15 s matches the retired issue #60 guard's window + /// and leaves margin over the measured attach delay. + static let transportTrustWindow: TimeInterval = 15 + + /// Whether a transport selection made `now` may be pinned (cached) + /// for the rest of the boot. Inside the window the connection is + /// still usable — the caller just must not reuse it, so the next + /// command re-derives the selection after the window closes. + /// An unknown launchd_sim identity keeps the marker-fallback + /// caching behavior: the race evidence only exists where the + /// probe works, and failing closed would rebuild on every command + /// in environments where it never does. + static func isTransportSelectionTrustworthy(token: HIDBootToken, now: Date) -> Bool { + guard let launchdSim = token.launchdSim else { return true } + return now.timeIntervalSince(launchdSim.startedAt) >= transportTrustWindow + } + /// The current boot token for a simulator. The probe is injectable /// so the composition is testable without live processes. static func token( diff --git a/Sources/iOSSimBackend/HID/HIDInteractor.swift b/Sources/iOSSimBackend/HID/HIDInteractor.swift index e27e4c14..9f0f625b 100644 --- a/Sources/iOSSimBackend/HID/HIDInteractor.swift +++ b/Sources/iOSSimBackend/HID/HIDInteractor.swift @@ -25,9 +25,14 @@ public struct HIDInteractor { // connection's mach port dies with that boot, and a send through a // dead port hangs or silently drops input (issue #55), so reuse // must be gated on the token before anything is sent. + // `transportTrusted` is false when the transport was auto-selected + // inside the boot-attach trust window (issue #67): the entry may + // serve the command that created it but must not be reused, so the + // next command re-derives the selection. private struct CachedConnection { let hid: FBSimulatorHID let bootToken: HIDBootToken + let transportTrusted: Bool } private static var hidConnections: [String: CachedConnection] = [:] @@ -166,14 +171,23 @@ public struct HIDInteractor { private static func getOrCreateHIDConnection(for simulator: FBSimulator, logger: SimUseLogger) async throws -> FBSimulatorHID { let currentToken = HIDBootIdentity.token(dataDirectory: simulator.dataDirectory, udid: simulator.udid) if let cached = hidConnections[simulator.udid] { - if HIDBootIdentity.isReusable(cachedToken: cached.bootToken, currentToken: currentToken) { + let sameBoot = HIDBootIdentity.isReusable(cachedToken: cached.bootToken, currentToken: currentToken) + if sameBoot && cached.transportTrusted { logger.info().log("Using existing HID connection for simulator \(simulator.udid)") return cached.hid } - // The simulator was re-booted (or the boot identity is - // unknowable) since the connection was made: the cached - // handle's mach port is dead and must not be sent through. - logger.info().log("Boot identity changed for simulator \(simulator.udid) (cached: \(cached.bootToken); current: \(currentToken)); discarding cached HID connection") + if sameBoot { + // Same boot, but the transport was auto-selected inside + // the boot-attach trust window (issue #67): dtuhidd may + // have appeared since the probe, so the selection must + // be re-derived rather than pinned for the whole boot. + logger.info().log("Cached HID connection for \(simulator.udid) was created inside the transport trust window; discarding it to re-derive the transport selection") + } else { + // The simulator was re-booted (or the boot identity is + // unknowable) since the connection was made: the cached + // handle's mach port is dead and must not be sent through. + logger.info().log("Boot identity changed for simulator \(simulator.udid) (cached: \(cached.bootToken); current: \(currentToken)); discarding cached HID connection") + } cached.hid.disconnect() hidConnections.removeValue(forKey: simulator.udid) } @@ -199,8 +213,17 @@ public struct HIDInteractor { // debug override is set. let hid = try FBSimulatorHID(for: simulator, transport: transportOverride) - hidConnections[simulator.udid] = CachedConnection(hid: hid, bootToken: currentToken) - logger.info().log("HID connection created and cached for simulator \(simulator.udid)") + // A forced transport cannot race dtuhidd's boot-time attach; + // only the auto-selection is window-gated. + let transportTrusted = transportOverride != nil + || HIDBootIdentity.isTransportSelectionTrustworthy(token: currentToken, now: Date()) + hidConnections[simulator.udid] = CachedConnection( + hid: hid, bootToken: currentToken, transportTrusted: transportTrusted) + if transportTrusted { + logger.info().log("HID connection created and cached for simulator \(simulator.udid)") + } else { + logger.info().log("HID connection created for simulator \(simulator.udid) inside the transport trust window (launchd_sim uptime < \(Int(HIDBootIdentity.transportTrustWindow)) s); the next command re-derives the transport selection") + } return hid } diff --git a/Sources/iOSSimBackend/Types/SimUseLogger.swift b/Sources/iOSSimBackend/Types/SimUseLogger.swift index cd80c898..e278ff26 100644 --- a/Sources/iOSSimBackend/Types/SimUseLogger.swift +++ b/Sources/iOSSimBackend/Types/SimUseLogger.swift @@ -15,8 +15,15 @@ public final class SimUseLogger: FBCompositeLogger { self.init(loggers: [systemLogger]) } + /// The default logger writes to no visible sink; SIM_USE_DEBUG=1 + /// turns on stderr (which the daemon redirects to its logfile), so + /// info-lines like the HID transport-selection signals become + /// visible in the field (issue #67). The daemon keeps the + /// environment it was spawned with — combine with a daemon restart + /// or SIM_USE_NO_DAEMON=1. public override convenience init() { - self.init(debugLogging: false, writeToStdErr: false) + let debug = ProcessInfo.processInfo.environment["SIM_USE_DEBUG"] == "1" + self.init(debugLogging: debug, writeToStdErr: debug) } public func makeDefault() { diff --git a/Tests/HIDBootIdentityTests.swift b/Tests/HIDBootIdentityTests.swift index 8127cbfa..d6bd8969 100644 --- a/Tests/HIDBootIdentityTests.swift +++ b/Tests/HIDBootIdentityTests.swift @@ -88,6 +88,57 @@ struct HIDBootIdentityReusableTests { } } +// Upstream's HID transport auto-selection probes the simulator's +// process tree for dtuhidd at `FBSimulatorHID` construction — but +// dtuhidd attaches 0–3 s *after* boot on a simulator booted while +// Device Hub is open (issue #67). A selection made inside that window +// resolves to Indigo on a dtuhidd-suppressed simulator and, once +// cached, silently pins the dead transport for the whole boot (Indigo +// reports success without delivering, so recovery never fires). +// `isTransportSelectionTrustworthy` gates the cache: a connection +// built inside the window may be used once but must not be reused. + +@Suite("HIDBootIdentity.isTransportSelectionTrustworthy") +struct HIDTransportTrustWindowTests { + private let now = Date(timeIntervalSince1970: 1_785_000_000) + + private func token(uptime: TimeInterval?) -> HIDBootToken { + HIDBootToken( + launchdSim: uptime.map { LaunchdSimIdentity(pid: 74691, startedAt: now.addingTimeInterval(-$0)) }, + markerModificationDate: nil + ) + } + + @Test("A selection made inside the boot-attach window is not trustworthy") + func insideWindowIsNotTrustworthy() { + // dtuhidd was measured attaching 0–3 s after launchd_sim; the + // window leaves margin over that. Anything probed before the + // boundary may have raced the attach. + #expect(!HIDBootIdentity.isTransportSelectionTrustworthy(token: token(uptime: 2), now: now)) + #expect(!HIDBootIdentity.isTransportSelectionTrustworthy(token: token(uptime: 14.9), now: now)) + } + + @Test("A selection at or past the window boundary is trustworthy") + func atOrPastWindowIsTrustworthy() { + #expect(HIDBootIdentity.isTransportSelectionTrustworthy( + token: token(uptime: HIDBootIdentity.transportTrustWindow), now: now)) + #expect(HIDBootIdentity.isTransportSelectionTrustworthy(token: token(uptime: 300), now: now)) + } + + @Test("Unknown launchd_sim identity keeps the marker-fallback caching behavior") + func unknownIdentityIsTrustworthy() { + // The race evidence only exists where the launchd_sim probe + // works; failing closed here would rebuild the connection on + // every command in environments where the probe never works. + #expect(HIDBootIdentity.isTransportSelectionTrustworthy(token: token(uptime: nil), now: now)) + } + + @Test("A start time in the future (clock anomaly) is not trustworthy") + func futureStartTimeIsNotTrustworthy() { + #expect(!HIDBootIdentity.isTransportSelectionTrustworthy(token: token(uptime: -30), now: now)) + } +} + @Suite("HIDBootIdentity.token") struct HIDBootIdentityTokenTests { From 3f236b2039404dfac03112cad008017d3141b7b3 Mon Sep 17 00:00:00 2001 From: onevcat Date: Tue, 28 Jul 2026 15:42:31 +0900 Subject: [PATCH 2/2] fix: SIM_USE_DEBUG=1 now reaches every logger construction path Review follow-up: the env check lived only in SimUseLogger's no-arg initializer, so `ios batch` - which passes its own flags via SimUseLogger(writeToStdErr: verbose) - ignored SIM_USE_DEBUG entirely and the transport-selection diagnostics stayed invisible without --verbose. OR-merge the env switch inside the parameterized convenience initializer instead: one implementation covers every construction path, batch needs no change, and --verbose keeps its exact behavior. The env predicate takes an injectable environment for tests, which lock the SIM_USE_NO_DAEMON-style "literal 1 only" convention. Verified live: SIM_USE_DEBUG=1 sim-use ios batch --step "sleep 0.1" (no --verbose) now emits the diagnostic info-lines on stderr; without the env var stderr stays quiet. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Rir4WwrPShf5DYzziJ8RyC Signed-off-by: onevcat --- CHANGELOG.md | 2 +- .../iOSSimBackend/Types/SimUseLogger.swift | 32 ++++++++++++------- Tests/SimUseLoggerTests.swift | 30 +++++++++++++++++ 3 files changed, 52 insertions(+), 12 deletions(-) create mode 100644 Tests/SimUseLoggerTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index e20206f2..c3fde11a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - The agent-eval suite can now pin exactly which sim-use binary a run evaluates: `make eval ARGS="-b "` / `scripts/eval.sh --sim-use ` / `run.py --sim-use ` (default remains whatever `sim-use` resolves to on PATH). The wrapper and runner print `sim-use under test: ()` up front and the report header records it, so a run can never silently exercise the wrong binary — and development builds under `.build/` can be evaluated directly. New repo skill `.claude/skills/run-evals/` orchestrates the whole flow for agents and contributors: environment prep (Device Hub closed, fixtures installed), binary selection, cost confirmation, and verdict triage; `e2e/agent-evals/README.md` documents the new prereqs and flags. - `SIM_USE_HID_TRANSPORT=indigo|dtuhid` debug override to force a specific iOS HID transport (default: automatic per-boot selection). The per-UDID daemon keeps the environment it was spawned with — combine with `SIM_USE_NO_DAEMON=1` or restart the daemon for ad-hoc experiments. -- `SIM_USE_DEBUG=1` turns on the default logger's stderr sink (with debug-level detail), making internal info-lines — notably the HID transport-selection signals logged at connection creation — visible in normal runs and in the daemon logfile. Same daemon-environment caveat as `SIM_USE_HID_TRANSPORT`. (#67) +- `SIM_USE_DEBUG=1` turns on the logger's stderr sink (with debug-level detail) on every construction path, including `ios batch` without `--verbose`, making internal info-lines — notably the HID transport-selection signals logged at connection creation — visible in normal runs and in the daemon logfile. Same daemon-environment caveat as `SIM_USE_HID_TRANSPORT`. (#67) - `describe-ui --no-raw` (top-level, `ios describe-ui`, and `android describe-ui`): with `--json`, omit the raw accessibility tree from the envelope. `data.raw` typically dominates the payload on real app screens and is only useful for debugging sim-use itself; `outline` / `entries` / `lists` are unaffected. - *Keeping output small* section in `skills/sim-use/SKILL.md`: steers agents to prefer the text outline, pair `--json` with `--no-raw`, verify via outline instead of screenshots, reuse the verify read as the next observe, and batch known sequences. - Viewer: `GET /api/snapshot` now forwards the CLI's calibrated interface orientation as `screen.orientation` (omitted when `describe-ui` reports none — Android, legacy daemons), and the SPA tags a rotated screen next to the W×H readout, e.g. `874×402 (landscape-right)`. (#57) diff --git a/Sources/iOSSimBackend/Types/SimUseLogger.swift b/Sources/iOSSimBackend/Types/SimUseLogger.swift index e278ff26..c171a63a 100644 --- a/Sources/iOSSimBackend/Types/SimUseLogger.swift +++ b/Sources/iOSSimBackend/Types/SimUseLogger.swift @@ -7,23 +7,33 @@ public final class SimUseLogger: FBCompositeLogger { super.init(loggers: loggers) } + /// SIM_USE_DEBUG=1 forces stderr and debug-level output on every + /// logger regardless of what the call site asked for (OR-merged + /// below), so info-lines like the HID transport-selection signals + /// become visible in the field (issue #67) — including call sites + /// that pass their own flags, like `ios batch --verbose`. The + /// daemon redirects stderr to its logfile but keeps the + /// environment it was spawned with — combine with a daemon + /// restart or SIM_USE_NO_DAEMON=1. + static func debugEnvironmentEnabled( + _ environment: [String: String] = ProcessInfo.processInfo.environment + ) -> Bool { + environment["SIM_USE_DEBUG"] == "1" + } + public convenience init(debugLogging: Bool = false, writeToStdErr: Bool = true) { + let debug = Self.debugEnvironmentEnabled() let systemLogger = FBControlCoreLoggerFactory.systemLoggerWriting( - toStderr: writeToStdErr, - withDebugLogging: debugLogging + toStderr: writeToStdErr || debug, + withDebugLogging: debugLogging || debug ) self.init(loggers: [systemLogger]) } - - /// The default logger writes to no visible sink; SIM_USE_DEBUG=1 - /// turns on stderr (which the daemon redirects to its logfile), so - /// info-lines like the HID transport-selection signals become - /// visible in the field (issue #67). The daemon keeps the - /// environment it was spawned with — combine with a daemon restart - /// or SIM_USE_NO_DAEMON=1. + + /// The default logger writes to no visible sink unless + /// SIM_USE_DEBUG=1 turns one on (see `debugEnvironmentEnabled`). public override convenience init() { - let debug = ProcessInfo.processInfo.environment["SIM_USE_DEBUG"] == "1" - self.init(debugLogging: debug, writeToStdErr: debug) + self.init(debugLogging: false, writeToStdErr: false) } public func makeDefault() { diff --git a/Tests/SimUseLoggerTests.swift b/Tests/SimUseLoggerTests.swift new file mode 100644 index 00000000..fb8f257b --- /dev/null +++ b/Tests/SimUseLoggerTests.swift @@ -0,0 +1,30 @@ +// SPDX-License-Identifier: Apache-2.0 +@testable import iOSSimBackend +import Foundation +import Testing + +// SIM_USE_DEBUG=1 is the diagnostics switch for issue #67: it must +// reach every logger construction path (including call sites that +// pass their own flags, like `ios batch --verbose`), and it follows +// the SIM_USE_NO_DAEMON convention of matching the literal "1" only. + +@Suite("SimUseLogger.debugEnvironmentEnabled") +struct SimUseLoggerDebugEnvironmentTests { + + @Test("The literal value 1 enables debug output") + func literalOneEnables() { + #expect(SimUseLogger.debugEnvironmentEnabled(["SIM_USE_DEBUG": "1"])) + } + + @Test("Any other value is ignored") + func otherValuesAreIgnored() { + #expect(!SimUseLogger.debugEnvironmentEnabled(["SIM_USE_DEBUG": "0"])) + #expect(!SimUseLogger.debugEnvironmentEnabled(["SIM_USE_DEBUG": "true"])) + #expect(!SimUseLogger.debugEnvironmentEnabled(["SIM_USE_DEBUG": ""])) + } + + @Test("An unset variable leaves debug output off") + func unsetIsOff() { + #expect(!SimUseLogger.debugEnvironmentEnabled([:])) + } +}