diff --git a/CHANGELOG.md b/CHANGELOG.md index 9e903962..c3fde11a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - The agent-eval suite can now pin exactly which sim-use binary a run evaluates: `make eval ARGS="-b "` / `scripts/eval.sh --sim-use ` / `run.py --sim-use ` (default remains whatever `sim-use` resolves to on PATH). The wrapper and runner print `sim-use under test: ()` up front and the report header records it, so a run can never silently exercise the wrong binary — and development builds under `.build/` can be evaluated directly. New repo skill `.claude/skills/run-evals/` orchestrates the whole flow for agents and contributors: environment prep (Device Hub closed, fixtures installed), binary selection, cost confirmation, and verdict triage; `e2e/agent-evals/README.md` documents the new prereqs and flags. - `SIM_USE_HID_TRANSPORT=indigo|dtuhid` debug override to force a specific iOS HID transport (default: automatic per-boot selection). The per-UDID daemon keeps the environment it was spawned with — combine with `SIM_USE_NO_DAEMON=1` or restart the daemon for ad-hoc experiments. +- `SIM_USE_DEBUG=1` turns on the logger's stderr sink (with debug-level detail) on every construction path, including `ios batch` without `--verbose`, making internal info-lines — notably the HID transport-selection signals logged at connection creation — visible in normal runs and in the daemon logfile. Same daemon-environment caveat as `SIM_USE_HID_TRANSPORT`. (#67) - `describe-ui --no-raw` (top-level, `ios describe-ui`, and `android describe-ui`): with `--json`, omit the raw accessibility tree from the envelope. `data.raw` typically dominates the payload on real app screens and is only useful for debugging sim-use itself; `outline` / `entries` / `lists` are unaffected. - *Keeping output small* section in `skills/sim-use/SKILL.md`: steers agents to prefer the text outline, pair `--json` with `--no-raw`, verify via outline instead of screenshots, reuse the verify read as the next observe, and batch known sequences. - Viewer: `GET /api/snapshot` now forwards the CLI's calibrated interface orientation as `screen.orientation` (omitted when `describe-ui` reports none — Android, legacy daemons), and the SPA tags a rotated screen next to the W×H readout, e.g. `874×402 (landscape-right)`. (#57) @@ -26,6 +27,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- The iOS HID transport auto-selection no longer races dtuhidd's boot-time attach (#67). dtuhidd appears 0–3 s after boot on a simulator booted while Device Hub is open; a HID verb landing inside that window probed the process tree too early, auto-selected the legacy Indigo transport, and the cached connection pinned the wrong transport for the entire boot — `type` exited 0 while delivering nothing, with no error to trigger recovery. A connection whose transport was auto-selected while `launchd_sim` uptime is under 15 s is now served once but never reused, so the next command re-derives the selection after the attach window closes. Steady-state behavior (uptime ≥ 15 s at first HID verb) is unchanged. - Android `record-video` now preserves a static final frame through the requested stop time, and waits for stdout chunks already being delivered before finalizing the MP4. - `record-video` frame rate is no longer capped at the ~8–10 fps the screenshot-polling loop topped out at (iOS honors `--fps` up to 60 with smooth constant-rate playback; an Android emulator captured ~50 fps under motion). Recording no longer burns CPU decoding/re-encoding every frame. - Building the XCFrameworks now requires [XcodeGen](https://github.com/yonaskolb/XcodeGen) (`brew install xcodegen`) — the idb checkout generates `FBSimulatorControl.xcodeproj` from `project.yml`. The frameworks are built without library evolution, so `build_products/` is locked to the toolchain that produced it: re-run `./scripts/build.sh dev` after switching Xcode versions (CI keys its cache on the Xcode version for the same reason). diff --git a/README.md b/README.md index b5a4a630..4f57aebc 100644 --- a/README.md +++ b/README.md @@ -126,9 +126,15 @@ it), **including Device Hub workflows**: Hub was open) is driven through dtuhidd's CoreDevice HID service, and everything else through the legacy SimulatorKit path. No reboot dance, no guard errors — `tap` / `type` / `swipe` work in both states. +- A selection made within 15 s of boot (dtuhidd attaches 0–3 s after + boot, so a `simctl boot && sim-use type` script can probe too early) + is used once but not cached; the next command re-derives it, so an + early wrong pick never sticks for the boot. - `SIM_USE_HID_TRANSPORT=indigo|dtuhid` forces a specific transport for - debugging (combine with `SIM_USE_NO_DAEMON=1` — the per-UDID daemon - keeps the environment it was first spawned with). + debugging, and `SIM_USE_DEBUG=1` surfaces the transport-selection + signals (and other internal info-lines) on stderr / in the daemon + logfile (combine either with `SIM_USE_NO_DAEMON=1` — the per-UDID + daemon keeps the environment it was first spawned with). - Xcode 27 no longer bundles Simulator.app; the one from an Xcode 26.x install still works for viewing simulators, as does Device Hub itself. - Work record: `docs/ai/xxxx-xcode27-support/README.md`. diff --git a/Sources/iOSSimBackend/HID/HIDBootIdentity.swift b/Sources/iOSSimBackend/HID/HIDBootIdentity.swift index 41810d15..7a28874e 100644 --- a/Sources/iOSSimBackend/HID/HIDBootIdentity.swift +++ b/Sources/iOSSimBackend/HID/HIDBootIdentity.swift @@ -63,6 +63,29 @@ enum HIDBootIdentity { } } + /// The window after boot inside which upstream's HID transport + /// auto-selection cannot be trusted: dtuhidd attaches 0–3 s after + /// `launchd_sim` on a simulator booted while Device Hub is open + /// (issue #67), so a selection probed before it appears resolves + /// to Indigo on a dtuhidd-suppressed simulator — and Indigo + /// reports success without delivering, so no recovery path ever + /// rebuilds it. 15 s matches the retired issue #60 guard's window + /// and leaves margin over the measured attach delay. + static let transportTrustWindow: TimeInterval = 15 + + /// Whether a transport selection made `now` may be pinned (cached) + /// for the rest of the boot. Inside the window the connection is + /// still usable — the caller just must not reuse it, so the next + /// command re-derives the selection after the window closes. + /// An unknown launchd_sim identity keeps the marker-fallback + /// caching behavior: the race evidence only exists where the + /// probe works, and failing closed would rebuild on every command + /// in environments where it never does. + static func isTransportSelectionTrustworthy(token: HIDBootToken, now: Date) -> Bool { + guard let launchdSim = token.launchdSim else { return true } + return now.timeIntervalSince(launchdSim.startedAt) >= transportTrustWindow + } + /// The current boot token for a simulator. The probe is injectable /// so the composition is testable without live processes. static func token( diff --git a/Sources/iOSSimBackend/HID/HIDInteractor.swift b/Sources/iOSSimBackend/HID/HIDInteractor.swift index e27e4c14..9f0f625b 100644 --- a/Sources/iOSSimBackend/HID/HIDInteractor.swift +++ b/Sources/iOSSimBackend/HID/HIDInteractor.swift @@ -25,9 +25,14 @@ public struct HIDInteractor { // connection's mach port dies with that boot, and a send through a // dead port hangs or silently drops input (issue #55), so reuse // must be gated on the token before anything is sent. + // `transportTrusted` is false when the transport was auto-selected + // inside the boot-attach trust window (issue #67): the entry may + // serve the command that created it but must not be reused, so the + // next command re-derives the selection. private struct CachedConnection { let hid: FBSimulatorHID let bootToken: HIDBootToken + let transportTrusted: Bool } private static var hidConnections: [String: CachedConnection] = [:] @@ -166,14 +171,23 @@ public struct HIDInteractor { private static func getOrCreateHIDConnection(for simulator: FBSimulator, logger: SimUseLogger) async throws -> FBSimulatorHID { let currentToken = HIDBootIdentity.token(dataDirectory: simulator.dataDirectory, udid: simulator.udid) if let cached = hidConnections[simulator.udid] { - if HIDBootIdentity.isReusable(cachedToken: cached.bootToken, currentToken: currentToken) { + let sameBoot = HIDBootIdentity.isReusable(cachedToken: cached.bootToken, currentToken: currentToken) + if sameBoot && cached.transportTrusted { logger.info().log("Using existing HID connection for simulator \(simulator.udid)") return cached.hid } - // The simulator was re-booted (or the boot identity is - // unknowable) since the connection was made: the cached - // handle's mach port is dead and must not be sent through. - logger.info().log("Boot identity changed for simulator \(simulator.udid) (cached: \(cached.bootToken); current: \(currentToken)); discarding cached HID connection") + if sameBoot { + // Same boot, but the transport was auto-selected inside + // the boot-attach trust window (issue #67): dtuhidd may + // have appeared since the probe, so the selection must + // be re-derived rather than pinned for the whole boot. + logger.info().log("Cached HID connection for \(simulator.udid) was created inside the transport trust window; discarding it to re-derive the transport selection") + } else { + // The simulator was re-booted (or the boot identity is + // unknowable) since the connection was made: the cached + // handle's mach port is dead and must not be sent through. + logger.info().log("Boot identity changed for simulator \(simulator.udid) (cached: \(cached.bootToken); current: \(currentToken)); discarding cached HID connection") + } cached.hid.disconnect() hidConnections.removeValue(forKey: simulator.udid) } @@ -199,8 +213,17 @@ public struct HIDInteractor { // debug override is set. let hid = try FBSimulatorHID(for: simulator, transport: transportOverride) - hidConnections[simulator.udid] = CachedConnection(hid: hid, bootToken: currentToken) - logger.info().log("HID connection created and cached for simulator \(simulator.udid)") + // A forced transport cannot race dtuhidd's boot-time attach; + // only the auto-selection is window-gated. + let transportTrusted = transportOverride != nil + || HIDBootIdentity.isTransportSelectionTrustworthy(token: currentToken, now: Date()) + hidConnections[simulator.udid] = CachedConnection( + hid: hid, bootToken: currentToken, transportTrusted: transportTrusted) + if transportTrusted { + logger.info().log("HID connection created and cached for simulator \(simulator.udid)") + } else { + logger.info().log("HID connection created for simulator \(simulator.udid) inside the transport trust window (launchd_sim uptime < \(Int(HIDBootIdentity.transportTrustWindow)) s); the next command re-derives the transport selection") + } return hid } diff --git a/Sources/iOSSimBackend/Types/SimUseLogger.swift b/Sources/iOSSimBackend/Types/SimUseLogger.swift index cd80c898..c171a63a 100644 --- a/Sources/iOSSimBackend/Types/SimUseLogger.swift +++ b/Sources/iOSSimBackend/Types/SimUseLogger.swift @@ -7,14 +7,31 @@ public final class SimUseLogger: FBCompositeLogger { super.init(loggers: loggers) } + /// SIM_USE_DEBUG=1 forces stderr and debug-level output on every + /// logger regardless of what the call site asked for (OR-merged + /// below), so info-lines like the HID transport-selection signals + /// become visible in the field (issue #67) — including call sites + /// that pass their own flags, like `ios batch --verbose`. The + /// daemon redirects stderr to its logfile but keeps the + /// environment it was spawned with — combine with a daemon + /// restart or SIM_USE_NO_DAEMON=1. + static func debugEnvironmentEnabled( + _ environment: [String: String] = ProcessInfo.processInfo.environment + ) -> Bool { + environment["SIM_USE_DEBUG"] == "1" + } + public convenience init(debugLogging: Bool = false, writeToStdErr: Bool = true) { + let debug = Self.debugEnvironmentEnabled() let systemLogger = FBControlCoreLoggerFactory.systemLoggerWriting( - toStderr: writeToStdErr, - withDebugLogging: debugLogging + toStderr: writeToStdErr || debug, + withDebugLogging: debugLogging || debug ) self.init(loggers: [systemLogger]) } - + + /// The default logger writes to no visible sink unless + /// SIM_USE_DEBUG=1 turns one on (see `debugEnvironmentEnabled`). public override convenience init() { self.init(debugLogging: false, writeToStdErr: false) } diff --git a/Tests/HIDBootIdentityTests.swift b/Tests/HIDBootIdentityTests.swift index 8127cbfa..d6bd8969 100644 --- a/Tests/HIDBootIdentityTests.swift +++ b/Tests/HIDBootIdentityTests.swift @@ -88,6 +88,57 @@ struct HIDBootIdentityReusableTests { } } +// Upstream's HID transport auto-selection probes the simulator's +// process tree for dtuhidd at `FBSimulatorHID` construction — but +// dtuhidd attaches 0–3 s *after* boot on a simulator booted while +// Device Hub is open (issue #67). A selection made inside that window +// resolves to Indigo on a dtuhidd-suppressed simulator and, once +// cached, silently pins the dead transport for the whole boot (Indigo +// reports success without delivering, so recovery never fires). +// `isTransportSelectionTrustworthy` gates the cache: a connection +// built inside the window may be used once but must not be reused. + +@Suite("HIDBootIdentity.isTransportSelectionTrustworthy") +struct HIDTransportTrustWindowTests { + private let now = Date(timeIntervalSince1970: 1_785_000_000) + + private func token(uptime: TimeInterval?) -> HIDBootToken { + HIDBootToken( + launchdSim: uptime.map { LaunchdSimIdentity(pid: 74691, startedAt: now.addingTimeInterval(-$0)) }, + markerModificationDate: nil + ) + } + + @Test("A selection made inside the boot-attach window is not trustworthy") + func insideWindowIsNotTrustworthy() { + // dtuhidd was measured attaching 0–3 s after launchd_sim; the + // window leaves margin over that. Anything probed before the + // boundary may have raced the attach. + #expect(!HIDBootIdentity.isTransportSelectionTrustworthy(token: token(uptime: 2), now: now)) + #expect(!HIDBootIdentity.isTransportSelectionTrustworthy(token: token(uptime: 14.9), now: now)) + } + + @Test("A selection at or past the window boundary is trustworthy") + func atOrPastWindowIsTrustworthy() { + #expect(HIDBootIdentity.isTransportSelectionTrustworthy( + token: token(uptime: HIDBootIdentity.transportTrustWindow), now: now)) + #expect(HIDBootIdentity.isTransportSelectionTrustworthy(token: token(uptime: 300), now: now)) + } + + @Test("Unknown launchd_sim identity keeps the marker-fallback caching behavior") + func unknownIdentityIsTrustworthy() { + // The race evidence only exists where the launchd_sim probe + // works; failing closed here would rebuild the connection on + // every command in environments where the probe never works. + #expect(HIDBootIdentity.isTransportSelectionTrustworthy(token: token(uptime: nil), now: now)) + } + + @Test("A start time in the future (clock anomaly) is not trustworthy") + func futureStartTimeIsNotTrustworthy() { + #expect(!HIDBootIdentity.isTransportSelectionTrustworthy(token: token(uptime: -30), now: now)) + } +} + @Suite("HIDBootIdentity.token") struct HIDBootIdentityTokenTests { diff --git a/Tests/SimUseLoggerTests.swift b/Tests/SimUseLoggerTests.swift new file mode 100644 index 00000000..fb8f257b --- /dev/null +++ b/Tests/SimUseLoggerTests.swift @@ -0,0 +1,30 @@ +// SPDX-License-Identifier: Apache-2.0 +@testable import iOSSimBackend +import Foundation +import Testing + +// SIM_USE_DEBUG=1 is the diagnostics switch for issue #67: it must +// reach every logger construction path (including call sites that +// pass their own flags, like `ios batch --verbose`), and it follows +// the SIM_USE_NO_DAEMON convention of matching the literal "1" only. + +@Suite("SimUseLogger.debugEnvironmentEnabled") +struct SimUseLoggerDebugEnvironmentTests { + + @Test("The literal value 1 enables debug output") + func literalOneEnables() { + #expect(SimUseLogger.debugEnvironmentEnabled(["SIM_USE_DEBUG": "1"])) + } + + @Test("Any other value is ignored") + func otherValuesAreIgnored() { + #expect(!SimUseLogger.debugEnvironmentEnabled(["SIM_USE_DEBUG": "0"])) + #expect(!SimUseLogger.debugEnvironmentEnabled(["SIM_USE_DEBUG": "true"])) + #expect(!SimUseLogger.debugEnvironmentEnabled(["SIM_USE_DEBUG": ""])) + } + + @Test("An unset variable leaves debug output off") + func unsetIsOff() { + #expect(!SimUseLogger.debugEnvironmentEnabled([:])) + } +}