Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

- `sim-use ios-device` (experimental): drive a development-signed app on a physical iPhone or iPad. `devices` lists attached devices, `ui` prints the foreground app's accessibility tree, and `tap --label` / `--label-contains` sends Activate to one unambiguous match. sim-use installs and signs no runner and needs no Developer Disk Image; the device must be unlocked and the target app must have `get-task-allow=true`. This channel intentionally omits coordinate tap, swipe and gesture because the daemon exposes no element geometry.
- `sim-use ios-device ui` now renders each element's accessibility identifier as `#id`, and `sim-use ios-device tap` accepts it as a positional `#<id>` or `--id` (mirroring the simulator tap). This is a stable handle to prefer when a label is dynamic — a navigation-bar back button is labelled with the previous screen's title but keeps `#BackButton`. The `@N` alias and coordinate forms remain unavailable on this channel (handles expire between processes; the daemon exposes no geometry). Label and identifier matching go through the same case-sensitive `SelectorTextMatcher` policy the simulator and Android surfaces already use, so a selector behaves identically across all three.
- `sim-use ios-device screenshot`: capture a PNG of a connected iPhone or iPad display. Capture runs over CoreDevice (`xcrun devicectl device capture screenshot`) rather than the accessibility audit channel, so — unlike `ui` and `tap` — it is not limited to development-signed foreground apps: whatever is on screen is captured, SpringBoard and system apps included. Device selection matches the other `ios-device` verbs (`--device` optional with exactly one attached), and `--output` follows the shared path semantics with a `Device Screenshot - <device name> - <timestamp>.png` default. A rejected path or a capture that fails mid-flight never removes an existing file at `--output`: the image lands in a temporary sibling and replaces the target only on success. The `--output` path resolution shared by the simulator screenshot and the video verbs is now factored into one `OutputFilePath` helper instead of two per-target copies.

### Fixed

Expand Down
13 changes: 10 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ All device-scoped commands accept `--device <ID>` (optional when only one simula
* **Top-level** — cross-platform verbs: `ui`, `tap`, `long-press`, `swipe`, `touch`, `multi-touch`, `type`, `paste`, `button`, `gesture`, `keyboard-state`, `screenshot`, `record-video`, `stream-video`, `app-state`. Same flags on iOS and Android.
* **`sim-use ios <verb>`** — iOS-only: `key`, `key-combo`, `key-sequence`, `batch`.
* **`sim-use android <verb>`** — Android-only: `init`, `devices`, `ping`, `scroll`.
* **`sim-use ios-device <verb>`** — physical iOS devices (experimental): `devices`, `ui`, `tap`. Separate from the top-level verbs because the capabilities differ — see [Physical iOS devices](#physical-ios-devices).
* **`sim-use ios-device <verb>`** — physical iOS devices (experimental): `devices`, `ui`, `screenshot`, `tap`. Separate from the top-level verbs because the capabilities differ — see [Physical iOS devices](#physical-ios-devices).

Run `sim-use --help` or `sim-use <command> --help` for the full flag set.

Expand Down Expand Up @@ -423,6 +423,12 @@ sim-use ios-device tap --label-contains "Reply" --element-type Button
# Or target the stable accessibility identifier shown as #id — the same
# `#id` positional the simulator tap accepts (--id works too).
sim-use ios-device tap '#BackButton'

# Screenshot — any screen, not limited to development-signed apps. Prints the
# absolute saved path on stdout (plus a confirmation on stderr).
sim-use ios-device screenshot
# /Users/me/Device Screenshot - My iPhone - 2026-08-27 at 09.34.10.png
sim-use ios-device screenshot --output shot.png
```

A device is addressed by UDID or ECID, and `--device` is optional only when exactly one is attached. Run `ui` again after every action: accessibility actions are fire-and-forget, so the follow-up read is the authoritative verification.
Expand All @@ -433,14 +439,15 @@ This channel deliberately differs from the simulator backend:

* **No element geometry.** There is no coordinate tap, `swipe`, `gesture` or `multi-touch`. Only the exposed `tap` accessibility action is currently supported; unsupported simulator verbs are not routed here.
* **No `@N` aliases, but stable `#id`s.** Element handles encode a live pointer and expire with their DTX connection, so — like the missing geometry — the cross-invocation `@N` alias cannot be backed faithfully and the outline advertises none. The stable accessibility identifier *can*: the outline shows each element's `#id`, and `tap` accepts it as a positional `#<id>` or `--id` (mirroring the simulator), alongside `--label` / `--label-contains` / `--element-type`. Prefer the `#id` when a label is dynamic — a navigation-bar back button is labelled with the previous screen's title but keeps `#BackButton`, and is an ordinary, tappable row in the outline.
* **Text output only.** The experimental `ios-device` commands do not yet support `--json`, screenshot or recording.
* **Screenshots go over CoreDevice, not the audit daemon.** `screenshot` shells out to `xcrun devicectl device capture screenshot`, a separate channel with different rules: it is not limited to development-signed foreground apps and captures whatever is on screen, SpringBoard and system apps included. Screen *recording* exists on the same channel (`devicectl device capture screen-record`) but is capability-gated per device (CoreDevice can report "Screen Recording is not supported by this device") and is not exposed yet.
* **Text output only.** The experimental `ios-device` commands do not yet support `--json` or recording.
* **Slower snapshots.** A full tree costs a few seconds. `ui --fast` stops at labelled elements and is roughly 40% quicker, at the cost of about a quarter of the elements.
* **Reading order, not screen order.** With no frames to sort by, the outline follows accessibility nesting and reading order.


## Architecture

sim-use drives iOS Simulators through the lower-level XCFrameworks of Facebook's [idb](https://github.com/facebook/idb) (statically linked), Apple's Accessibility APIs, and the simulator HID pipeline. Android devices are driven through an on-device bridge APK that exposes the AccessibilityService tree and input injection over HTTP, tunnelled via `adb forward`. Physical iOS devices go through a third path: idb's `FBDeviceControl` opens a lockdown service connection, over which sim-use speaks Apple's DTX message protocol to the accessibility audit daemon. Everything ships as a single binary. The established simulator and Android surfaces support `--json`; the experimental `ios-device` commands currently emit text only.
sim-use drives iOS Simulators through the lower-level XCFrameworks of Facebook's [idb](https://github.com/facebook/idb) (statically linked), Apple's Accessibility APIs, and the simulator HID pipeline. Android devices are driven through an on-device bridge APK that exposes the AccessibilityService tree and input injection over HTTP, tunnelled via `adb forward`. Physical iOS devices go through a third path: idb's `FBDeviceControl` opens a lockdown service connection, over which sim-use speaks Apple's DTX message protocol to the accessibility audit daemon (screen capture instead shells out to Xcode's `devicectl`). Everything ships as a single binary. The established simulator and Android surfaces support `--json`; the experimental `ios-device` commands currently emit text only.


## Viewer
Expand Down
99 changes: 99 additions & 0 deletions Sources/SimUseCore/OutputFilePath.swift
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
// SPDX-License-Identifier: Apache-2.0
import Foundation

/// Output-path resolution shared by every file-producing verb (screenshots,
/// video recordings) across platform backends, so `--output` behaves the same
/// everywhere: trim and tilde-expand the supplied path, anchor relative paths
/// at the current directory, treat an existing directory as a destination
/// for a default-named file, create missing parent directories, and replace
/// an existing file.
///
/// Resolution and preparation are deliberately separate steps: `resolve`
/// never touches the filesystem beyond read-only stats, so a caller can
/// validate the resolved URL (e.g. enforce an extension) and reject it
/// without having destroyed an existing file at the target.
public enum OutputFilePath {
/// Resolve the user-supplied `--output` argument into a concrete file
/// URL. `defaultFilename` is consulted when no path is supplied or when
/// the path names an existing directory; it is invoked at most once per
/// call so timestamped names stay consistent. Performs no filesystem
/// mutation — call `prepare(_:)` before writing to the returned URL.
public static func resolve(output: String?, defaultFilename: () -> String) -> URL {
let fileManager = FileManager.default
var cachedDefault: String?
func defaultName() -> String {
if let cachedDefault { return cachedDefault }
let name = defaultFilename()
cachedDefault = name
return name
}

let providedPath = output?.trimmingCharacters(in: .whitespacesAndNewlines)
let resolvedPath: String
if let providedPath, !providedPath.isEmpty {
resolvedPath = (providedPath as NSString).expandingTildeInPath
} else {
resolvedPath = defaultName()
}

let baseURL: URL
if resolvedPath.hasPrefix("/") {
baseURL = URL(fileURLWithPath: resolvedPath)
} else {
baseURL = URL(fileURLWithPath: fileManager.currentDirectoryPath).appendingPathComponent(resolvedPath)
}

var isDirectory: ObjCBool = false
if fileManager.fileExists(atPath: baseURL.path, isDirectory: &isDirectory), isDirectory.boolValue {
return baseURL.appendingPathComponent(defaultName())
}

return baseURL
}

/// Create the resolved URL's missing parent directories. Non-destructive;
/// safe to run before a capture that may still fail.
public static func createParentDirectory(for url: URL) throws {
let fileManager = FileManager.default
let directoryURL = url.deletingLastPathComponent()
if !fileManager.fileExists(atPath: directoryURL.path) {
try fileManager.createDirectory(at: directoryURL, withIntermediateDirectories: true, attributes: nil)
}
}

/// Destructive preparation of a resolved output URL: create missing
/// parent directories and remove an existing file at the target so the
/// subsequent write replaces it. Callers whose payload production can
/// still fail after this point should instead write to a temporary
/// sibling and replace on success, so a failed capture cannot destroy
/// the existing file.
public static func prepare(_ url: URL) throws {
let fileManager = FileManager.default
try createParentDirectory(for: url)

var isDirectory: ObjCBool = false
if fileManager.fileExists(atPath: url.path, isDirectory: &isDirectory) {
if isDirectory.boolValue {
throw CLIError(errorDescription: "Output path \(url.path) is a directory. Provide a file name or point to a different location.")
}
try fileManager.removeItem(at: url)
}
}

/// Collapse a free-form name (e.g. a user-editable device name) into a
/// single path component for default filenames: path separators become
/// "-" so a name like "My iPhone/Work" cannot introduce directory
/// hierarchy — or, via "..", escape the target directory — when the name
/// is embedded in a default filename.
public static func safeFilenameComponent(_ name: String) -> String {
name.replacingOccurrences(of: "/", with: "-")
}

/// Timestamp format shared by every screenshot default filename so paired
/// screenshots from cross-platform sessions sort together.
public static func screenshotTimestamp(_ date: Date) -> String {
let formatter = DateFormatter()
formatter.dateFormat = "yyyy-MM-dd 'at' HH.mm.ss"
return formatter.string(from: date)
}
}
51 changes: 7 additions & 44 deletions Sources/SimUseVideo/VideoOutputFile.swift
Original file line number Diff line number Diff line change
Expand Up @@ -8,51 +8,14 @@ public enum VideoOutputFile {
/// Resolve the user-supplied `--output` argument into a concrete
/// output file URL (default naming uses `fileExtension`). Every
/// platform backend and the cross-platform forwarder use the same
/// path semantics.
/// path semantics (`OutputFilePath`).
public static func prepareOutputURL(output: String?, fileExtension: String = "mp4") throws -> URL {
let fileManager = FileManager.default
let formatter = ISO8601DateFormatter()
formatter.formatOptions = [.withInternetDateTime]

let providedPath = output?.trimmingCharacters(in: .whitespacesAndNewlines)
let resolvedPath: String
if let providedPath, !providedPath.isEmpty {
resolvedPath = (providedPath as NSString).expandingTildeInPath
} else {
resolvedPath = "sim-use-video-\(formatter.string(from: Date())).\(fileExtension)"
}

let baseURL: URL
if resolvedPath.hasPrefix("/") {
baseURL = URL(fileURLWithPath: resolvedPath)
} else {
baseURL = URL(fileURLWithPath: fileManager.currentDirectoryPath).appendingPathComponent(resolvedPath)
}

var isDirectory: ObjCBool = false
if fileManager.fileExists(atPath: baseURL.path, isDirectory: &isDirectory), isDirectory.boolValue {
let filename = "sim-use-video-\(formatter.string(from: Date())).\(fileExtension)"
let directoryURL = baseURL
if !fileManager.fileExists(atPath: directoryURL.path) {
try fileManager.createDirectory(at: directoryURL, withIntermediateDirectories: true, attributes: nil)
}
return directoryURL.appendingPathComponent(filename)
let url = OutputFilePath.resolve(output: output) {
let formatter = ISO8601DateFormatter()
formatter.formatOptions = [.withInternetDateTime]
return "sim-use-video-\(formatter.string(from: Date())).\(fileExtension)"
}

let directoryURL = baseURL.deletingLastPathComponent()
if !fileManager.fileExists(atPath: directoryURL.path) {
try fileManager.createDirectory(at: directoryURL, withIntermediateDirectories: true, attributes: nil)
}

if fileManager.fileExists(atPath: baseURL.path) {
var existingIsDirectory: ObjCBool = false
fileManager.fileExists(atPath: baseURL.path, isDirectory: &existingIsDirectory)
if existingIsDirectory.boolValue {
throw CLIError(errorDescription: "Output path \(baseURL.path) is a directory. Provide a file name or point to a different location.")
}
try fileManager.removeItem(at: baseURL)
}

return baseURL
try OutputFilePath.prepare(url)
return url
}
}
78 changes: 78 additions & 0 deletions Sources/iOSDeviceBackend/Devicectl/Devicectl.swift
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
// SPDX-License-Identifier: Apache-2.0
import Foundation

/// Shells out to `xcrun devicectl` (CoreDevice) for capabilities the
/// accessibility audit channel does not offer — currently screen capture.
///
/// Unlike the audit channel, CoreDevice capture is not limited to
/// development-signed foreground apps: it captures whatever is on screen,
/// including SpringBoard and system apps. Device *selection* still goes
/// through `DeviceSession.resolveDevice` so every `ios-device` verb sees the
/// same device set and errors; only the capture itself runs over CoreDevice.
enum Devicectl {
struct Failure: Error, LocalizedError, CustomStringConvertible {
let message: String
var errorDescription: String? { message }
var description: String { message }
}

/// Argument vector for a screenshot capture, separated from the spawn so
/// tests can pin the invocation without a device. `--quiet` suppresses
/// devicectl's own progress output; the verb prints its own confirmation.
static func screenshotArguments(deviceIdentifier: String, destination: URL) -> [String] {
[
"devicectl", "device", "capture", "screenshot",
"--device", deviceIdentifier,
"--destination", destination.path,
"--timeout", "30",
"--quiet",
]
}

/// Runs devicectl and fails with its stderr on a non-zero exit.
/// `executablePath` is injectable so tests can drive the drain and error
/// mapping against `/bin/sh`; production uses the default `xcrun`.
static func run(arguments: [String], executablePath: String = "/usr/bin/xcrun") throws {
let process = Process()
process.executableURL = URL(fileURLWithPath: executablePath)
process.arguments = arguments

let stdout = Pipe()
let stderr = Pipe()
process.standardOutput = stdout
process.standardError = stderr

// Drain both pipes while the child runs so a chatty error path cannot
// fill the ~64 KB pipe buffer and deadlock `waitUntilExit()`. Same
// drain as `SimctlDeviceLister.runSimctl` and `Adb.run`.
let bufferLock = NSLock()
var errBuffer = Data()
stdout.fileHandleForReading.readabilityHandler = { handle in
_ = handle.availableData
}
stderr.fileHandleForReading.readabilityHandler = { handle in
let chunk = handle.availableData
guard !chunk.isEmpty else { return }
bufferLock.lock(); errBuffer.append(chunk); bufferLock.unlock()
}

do {
try process.run()
} catch {
throw Failure(message: "could not spawn xcrun devicectl: \(error.localizedDescription)")
}
process.waitUntilExit()

stdout.fileHandleForReading.readabilityHandler = nil
stderr.fileHandleForReading.readabilityHandler = nil
bufferLock.lock()
errBuffer.append(stderr.fileHandleForReading.readDataToEndOfFile())
bufferLock.unlock()

guard process.terminationStatus == 0 else {
let err = (String(data: errBuffer, encoding: .utf8) ?? "")
.trimmingCharacters(in: .whitespacesAndNewlines)
throw Failure(message: "xcrun devicectl exited \(process.terminationStatus)\(err.isEmpty ? "" : ": \(err)")")
}
}
}
Loading
Loading