Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

- The agent-eval suite can now pin exactly which sim-use binary a run evaluates: `make eval ARGS="-b <path>"` / `scripts/eval.sh --sim-use <path>` / `run.py --sim-use <path>` (default remains whatever `sim-use` resolves to on PATH). The wrapper and runner print `sim-use under test: <real path> (<version>)` up front and the report header records it, so a run can never silently exercise the wrong binary — and development builds under `.build/` can be evaluated directly. New repo skill `.claude/skills/run-evals/` orchestrates the whole flow for agents and contributors: environment prep (Device Hub closed, fixtures installed), binary selection, cost confirmation, and verdict triage; `e2e/agent-evals/README.md` documents the new prereqs and flags.
- `SIM_USE_HID_TRANSPORT=indigo|dtuhid` debug override to force a specific iOS HID transport (default: automatic per-boot selection). The per-UDID daemon keeps the environment it was spawned with — combine with `SIM_USE_NO_DAEMON=1` or restart the daemon for ad-hoc experiments.
- `SIM_USE_DEBUG=1` turns on the logger's stderr sink (with debug-level detail) on every construction path, including `ios batch` without `--verbose`, making internal info-lines — notably the HID transport-selection signals logged at connection creation — visible in normal runs and in the daemon logfile. Same daemon-environment caveat as `SIM_USE_HID_TRANSPORT`. (#67)
- `describe-ui --no-raw` (top-level, `ios describe-ui`, and `android describe-ui`): with `--json`, omit the raw accessibility tree from the envelope. `data.raw` typically dominates the payload on real app screens and is only useful for debugging sim-use itself; `outline` / `entries` / `lists` are unaffected.
- *Keeping output small* section in `skills/sim-use/SKILL.md`: steers agents to prefer the text outline, pair `--json` with `--no-raw`, verify via outline instead of screenshots, reuse the verify read as the next observe, and batch known sequences.
- Viewer: `GET /api/snapshot` now forwards the CLI's calibrated interface orientation as `screen.orientation` (omitted when `describe-ui` reports none — Android, legacy daemons), and the SPA tags a rotated screen next to the W×H readout, e.g. `874×402 (landscape-right)`. (#57)
Expand All @@ -26,6 +27,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Fixed

- The iOS HID transport auto-selection no longer races dtuhidd's boot-time attach (#67). dtuhidd appears 0–3 s after boot on a simulator booted while Device Hub is open; a HID verb landing inside that window probed the process tree too early, auto-selected the legacy Indigo transport, and the cached connection pinned the wrong transport for the entire boot — `type` exited 0 while delivering nothing, with no error to trigger recovery. A connection whose transport was auto-selected while `launchd_sim` uptime is under 15 s is now served once but never reused, so the next command re-derives the selection after the attach window closes. Steady-state behavior (uptime ≥ 15 s at first HID verb) is unchanged.
- Android `record-video` now preserves a static final frame through the requested stop time, and waits for stdout chunks already being delivered before finalizing the MP4.
- `record-video` frame rate is no longer capped at the ~8–10 fps the screenshot-polling loop topped out at (iOS honors `--fps` up to 60 with smooth constant-rate playback; an Android emulator captured ~50 fps under motion). Recording no longer burns CPU decoding/re-encoding every frame.
- Building the XCFrameworks now requires [XcodeGen](https://github.com/yonaskolb/XcodeGen) (`brew install xcodegen`) — the idb checkout generates `FBSimulatorControl.xcodeproj` from `project.yml`. The frameworks are built without library evolution, so `build_products/` is locked to the toolchain that produced it: re-run `./scripts/build.sh dev` after switching Xcode versions (CI keys its cache on the Xcode version for the same reason).
Expand Down
10 changes: 8 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,9 +126,15 @@ it), **including Device Hub workflows**:
Hub was open) is driven through dtuhidd's CoreDevice HID service, and
everything else through the legacy SimulatorKit path. No reboot dance,
no guard errors — `tap` / `type` / `swipe` work in both states.
- A selection made within 15 s of boot (dtuhidd attaches 0–3 s after
boot, so a `simctl boot && sim-use type` script can probe too early)
is used once but not cached; the next command re-derives it, so an
early wrong pick never sticks for the boot.
- `SIM_USE_HID_TRANSPORT=indigo|dtuhid` forces a specific transport for
debugging (combine with `SIM_USE_NO_DAEMON=1` — the per-UDID daemon
keeps the environment it was first spawned with).
debugging, and `SIM_USE_DEBUG=1` surfaces the transport-selection
signals (and other internal info-lines) on stderr / in the daemon
logfile (combine either with `SIM_USE_NO_DAEMON=1` — the per-UDID
daemon keeps the environment it was first spawned with).
- Xcode 27 no longer bundles Simulator.app; the one from an Xcode 26.x
install still works for viewing simulators, as does Device Hub itself.
- Work record: `docs/ai/xxxx-xcode27-support/README.md`.
Expand Down
23 changes: 23 additions & 0 deletions Sources/iOSSimBackend/HID/HIDBootIdentity.swift
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,29 @@ enum HIDBootIdentity {
}
}

/// The window after boot inside which upstream's HID transport
/// auto-selection cannot be trusted: dtuhidd attaches 0–3 s after
/// `launchd_sim` on a simulator booted while Device Hub is open
/// (issue #67), so a selection probed before it appears resolves
/// to Indigo on a dtuhidd-suppressed simulator — and Indigo
/// reports success without delivering, so no recovery path ever
/// rebuilds it. 15 s matches the retired issue #60 guard's window
/// and leaves margin over the measured attach delay.
static let transportTrustWindow: TimeInterval = 15

/// Whether a transport selection made `now` may be pinned (cached)
/// for the rest of the boot. Inside the window the connection is
/// still usable — the caller just must not reuse it, so the next
/// command re-derives the selection after the window closes.
/// An unknown launchd_sim identity keeps the marker-fallback
/// caching behavior: the race evidence only exists where the
/// probe works, and failing closed would rebuild on every command
/// in environments where it never does.
static func isTransportSelectionTrustworthy(token: HIDBootToken, now: Date) -> Bool {
guard let launchdSim = token.launchdSim else { return true }
return now.timeIntervalSince(launchdSim.startedAt) >= transportTrustWindow
}

/// The current boot token for a simulator. The probe is injectable
/// so the composition is testable without live processes.
static func token(
Expand Down
37 changes: 30 additions & 7 deletions Sources/iOSSimBackend/HID/HIDInteractor.swift
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,14 @@ public struct HIDInteractor {
// connection's mach port dies with that boot, and a send through a
// dead port hangs or silently drops input (issue #55), so reuse
// must be gated on the token before anything is sent.
// `transportTrusted` is false when the transport was auto-selected
// inside the boot-attach trust window (issue #67): the entry may
// serve the command that created it but must not be reused, so the
// next command re-derives the selection.
private struct CachedConnection {
let hid: FBSimulatorHID
let bootToken: HIDBootToken
let transportTrusted: Bool
}

private static var hidConnections: [String: CachedConnection] = [:]
Expand Down Expand Up @@ -166,14 +171,23 @@ public struct HIDInteractor {
private static func getOrCreateHIDConnection(for simulator: FBSimulator, logger: SimUseLogger) async throws -> FBSimulatorHID {
let currentToken = HIDBootIdentity.token(dataDirectory: simulator.dataDirectory, udid: simulator.udid)
if let cached = hidConnections[simulator.udid] {
if HIDBootIdentity.isReusable(cachedToken: cached.bootToken, currentToken: currentToken) {
let sameBoot = HIDBootIdentity.isReusable(cachedToken: cached.bootToken, currentToken: currentToken)
if sameBoot && cached.transportTrusted {
logger.info().log("Using existing HID connection for simulator \(simulator.udid)")
return cached.hid
}
// The simulator was re-booted (or the boot identity is
// unknowable) since the connection was made: the cached
// handle's mach port is dead and must not be sent through.
logger.info().log("Boot identity changed for simulator \(simulator.udid) (cached: \(cached.bootToken); current: \(currentToken)); discarding cached HID connection")
if sameBoot {
// Same boot, but the transport was auto-selected inside
// the boot-attach trust window (issue #67): dtuhidd may
// have appeared since the probe, so the selection must
// be re-derived rather than pinned for the whole boot.
logger.info().log("Cached HID connection for \(simulator.udid) was created inside the transport trust window; discarding it to re-derive the transport selection")
} else {
// The simulator was re-booted (or the boot identity is
// unknowable) since the connection was made: the cached
// handle's mach port is dead and must not be sent through.
logger.info().log("Boot identity changed for simulator \(simulator.udid) (cached: \(cached.bootToken); current: \(currentToken)); discarding cached HID connection")
}
cached.hid.disconnect()
hidConnections.removeValue(forKey: simulator.udid)
}
Expand All @@ -199,8 +213,17 @@ public struct HIDInteractor {
// debug override is set.
let hid = try FBSimulatorHID(for: simulator, transport: transportOverride)

hidConnections[simulator.udid] = CachedConnection(hid: hid, bootToken: currentToken)
logger.info().log("HID connection created and cached for simulator \(simulator.udid)")
// A forced transport cannot race dtuhidd's boot-time attach;
// only the auto-selection is window-gated.
let transportTrusted = transportOverride != nil
|| HIDBootIdentity.isTransportSelectionTrustworthy(token: currentToken, now: Date())
hidConnections[simulator.udid] = CachedConnection(
hid: hid, bootToken: currentToken, transportTrusted: transportTrusted)
if transportTrusted {
logger.info().log("HID connection created and cached for simulator \(simulator.udid)")
} else {
logger.info().log("HID connection created for simulator \(simulator.udid) inside the transport trust window (launchd_sim uptime < \(Int(HIDBootIdentity.transportTrustWindow)) s); the next command re-derives the transport selection")
}

return hid
}
Expand Down
23 changes: 20 additions & 3 deletions Sources/iOSSimBackend/Types/SimUseLogger.swift
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,31 @@ public final class SimUseLogger: FBCompositeLogger {
super.init(loggers: loggers)
}

/// SIM_USE_DEBUG=1 forces stderr and debug-level output on every
/// logger regardless of what the call site asked for (OR-merged
/// below), so info-lines like the HID transport-selection signals
/// become visible in the field (issue #67) — including call sites
/// that pass their own flags, like `ios batch --verbose`. The
/// daemon redirects stderr to its logfile but keeps the
/// environment it was spawned with — combine with a daemon
/// restart or SIM_USE_NO_DAEMON=1.
static func debugEnvironmentEnabled(
_ environment: [String: String] = ProcessInfo.processInfo.environment
) -> Bool {
environment["SIM_USE_DEBUG"] == "1"
}

public convenience init(debugLogging: Bool = false, writeToStdErr: Bool = true) {
let debug = Self.debugEnvironmentEnabled()
let systemLogger = FBControlCoreLoggerFactory.systemLoggerWriting(
toStderr: writeToStdErr,
withDebugLogging: debugLogging
toStderr: writeToStdErr || debug,
withDebugLogging: debugLogging || debug
)
self.init(loggers: [systemLogger])
}


/// The default logger writes to no visible sink unless
/// SIM_USE_DEBUG=1 turns one on (see `debugEnvironmentEnabled`).
public override convenience init() {
self.init(debugLogging: false, writeToStdErr: false)
}
Expand Down
51 changes: 51 additions & 0 deletions Tests/HIDBootIdentityTests.swift
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,57 @@ struct HIDBootIdentityReusableTests {
}
}

// Upstream's HID transport auto-selection probes the simulator's
// process tree for dtuhidd at `FBSimulatorHID` construction — but
// dtuhidd attaches 0–3 s *after* boot on a simulator booted while
// Device Hub is open (issue #67). A selection made inside that window
// resolves to Indigo on a dtuhidd-suppressed simulator and, once
// cached, silently pins the dead transport for the whole boot (Indigo
// reports success without delivering, so recovery never fires).
// `isTransportSelectionTrustworthy` gates the cache: a connection
// built inside the window may be used once but must not be reused.

@Suite("HIDBootIdentity.isTransportSelectionTrustworthy")
struct HIDTransportTrustWindowTests {
private let now = Date(timeIntervalSince1970: 1_785_000_000)

private func token(uptime: TimeInterval?) -> HIDBootToken {
HIDBootToken(
launchdSim: uptime.map { LaunchdSimIdentity(pid: 74691, startedAt: now.addingTimeInterval(-$0)) },
markerModificationDate: nil
)
}

@Test("A selection made inside the boot-attach window is not trustworthy")
func insideWindowIsNotTrustworthy() {
// dtuhidd was measured attaching 0–3 s after launchd_sim; the
// window leaves margin over that. Anything probed before the
// boundary may have raced the attach.
#expect(!HIDBootIdentity.isTransportSelectionTrustworthy(token: token(uptime: 2), now: now))
#expect(!HIDBootIdentity.isTransportSelectionTrustworthy(token: token(uptime: 14.9), now: now))
}

@Test("A selection at or past the window boundary is trustworthy")
func atOrPastWindowIsTrustworthy() {
#expect(HIDBootIdentity.isTransportSelectionTrustworthy(
token: token(uptime: HIDBootIdentity.transportTrustWindow), now: now))
#expect(HIDBootIdentity.isTransportSelectionTrustworthy(token: token(uptime: 300), now: now))
}

@Test("Unknown launchd_sim identity keeps the marker-fallback caching behavior")
func unknownIdentityIsTrustworthy() {
// The race evidence only exists where the launchd_sim probe
// works; failing closed here would rebuild the connection on
// every command in environments where the probe never works.
#expect(HIDBootIdentity.isTransportSelectionTrustworthy(token: token(uptime: nil), now: now))
}

@Test("A start time in the future (clock anomaly) is not trustworthy")
func futureStartTimeIsNotTrustworthy() {
#expect(!HIDBootIdentity.isTransportSelectionTrustworthy(token: token(uptime: -30), now: now))
}
}

@Suite("HIDBootIdentity.token")
struct HIDBootIdentityTokenTests {

Expand Down
30 changes: 30 additions & 0 deletions Tests/SimUseLoggerTests.swift
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
// SPDX-License-Identifier: Apache-2.0
@testable import iOSSimBackend
import Foundation
import Testing

// SIM_USE_DEBUG=1 is the diagnostics switch for issue #67: it must
// reach every logger construction path (including call sites that
// pass their own flags, like `ios batch --verbose`), and it follows
// the SIM_USE_NO_DAEMON convention of matching the literal "1" only.

@Suite("SimUseLogger.debugEnvironmentEnabled")
struct SimUseLoggerDebugEnvironmentTests {

@Test("The literal value 1 enables debug output")
func literalOneEnables() {
#expect(SimUseLogger.debugEnvironmentEnabled(["SIM_USE_DEBUG": "1"]))
}

@Test("Any other value is ignored")
func otherValuesAreIgnored() {
#expect(!SimUseLogger.debugEnvironmentEnabled(["SIM_USE_DEBUG": "0"]))
#expect(!SimUseLogger.debugEnvironmentEnabled(["SIM_USE_DEBUG": "true"]))
#expect(!SimUseLogger.debugEnvironmentEnabled(["SIM_USE_DEBUG": ""]))
}

@Test("An unset variable leaves debug output off")
func unsetIsOff() {
#expect(!SimUseLogger.debugEnvironmentEnabled([:]))
}
}
Loading