From 14bbaa1d8be4130ffa1c4dc02ec5b1a7642fecdf Mon Sep 17 00:00:00 2001 From: onevcat Date: Wed, 29 Jul 2026 11:04:15 +0900 Subject: [PATCH 1/2] feat: Android support for stream-video + top-level cross-platform verb Closes #78. AndroidStreamVideoCommand serves two engines behind one flag surface: --format h264 pipes `adb exec-out screenrecord --output-format=h264 -` byte-for-byte to stdout (VFR, segment-restart loop from record-video minus the muxer; each new segment re-emits SPS/PPS, which ffplay/ffmpeg accept mid-stream), while mjpeg/raw/ffmpeg run a screencap JPEG loop with byte-identical framing to the iOS formats. The new top-level StreamVideo forwarder routes by UDID shape: bgra stays iOS-only, h264 is Android-only for now (iOS passthrough is a separate follow-up), and mismatches fail with a pointer to the right alternative. The 0.5.x agent-typo redirect shim drops stream-video from its list now that the verb is genuinely top-level. Streaming writes go through a POSIX-write stdout sink that ignores SIGPIPE and reports consumer hangup (ffplay quit, `head -c` done) as an orderly end-of-stream instead of crashing mid-summary. SIM_USE_SCREENRECORD_TIME_LIMIT= (debug) forces short screenrecord segments so the restart path is exercisable on API >= 34 devices, where `--time-limit 0` would otherwise never restart; it applies to Android record-video and stream-video alike. Verified: 1221 unit tests green; live AndroidStreamVideoTests E2E suite (5 tests: h264 Annex B smoke, mjpeg framing, raw length prefix, forced segment restart, consumer hangup) green against emulator-5554; ffmpeg decodes the live top-level h264 stream; iOS mjpeg via top-level streams 16 frames from a booted simulator. Co-Authored-By: Claude Fable 5 Signed-off-by: onevcat --- AGENTS.md | 4 +- CHANGELOG.md | 1 + README.md | 15 +- .../AndroidBackend/Verbs/AndroidCommand.swift | 1 + .../Verbs/AndroidRecordVideoCommand.swift | 31 +- .../Verbs/AndroidStreamVideoCommand.swift | 411 ++++++++++++++++++ Sources/SimUse/Commands/StreamVideo.swift | 177 ++++++++ Sources/SimUse/main.swift | 14 +- .../SimUseVideo/VideoRecordingOptions.swift | 17 + .../Verbs/IOSSimStreamVideoCommand.swift | 25 +- .../AndroidRecordVideoArgumentTests.swift | 21 + Tests/AndroidStreamVideoTests.swift | 145 ++++++ Tests/ForwarderInitializationGuardTests.swift | 5 + Tests/StreamVideoFormatMappingTests.swift | 50 +++ scripts/test-runner-android.sh | 1 + 15 files changed, 878 insertions(+), 40 deletions(-) create mode 100644 Sources/AndroidBackend/Verbs/AndroidStreamVideoCommand.swift create mode 100644 Sources/SimUse/Commands/StreamVideo.swift create mode 100644 Tests/AndroidStreamVideoTests.swift create mode 100644 Tests/StreamVideoFormatMappingTests.swift diff --git a/AGENTS.md b/AGENTS.md index 95a61b4a..5d16ac2f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -75,7 +75,7 @@ Five SwiftPM targets; dependency graph flows in one direction. | `SimUseCore` | `Sources/SimUseCore/` | Foundation + ArgumentParser | | `SimUseVideo` | `Sources/SimUseVideo/` | SimUseCore + AVFoundation/ImageIO | | `iOSSimBackend` | `Sources/iOSSimBackend/` | SimUseCore + SimUseVideo + FB* XCFrameworks + AVFoundation | -| `AndroidBackend` | `Sources/AndroidBackend/` | SimUseCore + ArgumentParser | +| `AndroidBackend` | `Sources/AndroidBackend/` | SimUseCore + SimUseVideo + ArgumentParser | | `SimUse` (executable) | `Sources/SimUse/` | SimUseCore + SimUseVideo + iOSSimBackend + AndroidBackend + FB* | `SimUseVideo` holds the platform-neutral host-side video plumbing (H.264 Annex B parsing, passthrough muxing, `AVAssetWriter` encoding, frame utilities) shared by the iOS and Android recording/streaming paths. It must stay FB*-free — anything that needs FBSimulatorControl belongs in `iOSSimBackend` (e.g. the `VideoFrameUtilities.captureScreenshotData` extension), anything adb-shaped in `AndroidBackend`. @@ -88,7 +88,7 @@ A verb (tap, swipe, type, ...) reaches three surfaces: 2. **`sim-use ios `** — `Sources/iOSSimBackend/Verbs/IOSSimCommand.swift`. 3. **`sim-use android `** — `Sources/AndroidBackend/Verbs/AndroidCommand.swift`. -Five verbs are iOS-only (`key`, `key-combo`, `key-sequence`, `stream-video`, `batch`) — no top-level alias. +Four verbs are iOS-only (`key`, `key-combo`, `key-sequence`, `batch`) — no top-level alias. ### Adding a new verb diff --git a/CHANGELOG.md b/CHANGELOG.md index 00c15180..bef35ecc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- Android support for `stream-video`, and a top-level cross-platform `stream-video` verb (#78). `sim-use android stream-video --format h264` streams native `adb screenrecord` H.264 Annex B passthrough to stdout (variable frame rate, survives screenrecord's per-invocation time limit by restarting mid-stream — each new segment re-emits SPS/PPS, which ffplay/ffmpeg accept); `mjpeg` / `raw` / `ffmpeg` provide a screencap-based JPEG loop (~7–8 FPS ceiling) with byte-identical framing to the iOS formats. The new top-level `sim-use stream-video` routes by UDID shape: `bgra` stays iOS-only, `h264` is Android-only for now, and the platform-mismatch cases fail with a pointer to the right alternative. Streaming now also treats consumer hangup (ffplay quit, `head -c` done) as an orderly end-of-stream on Android instead of dying on SIGPIPE. `SIM_USE_SCREENRECORD_TIME_LIMIT=` (debug) forces short screenrecord segments so the restart path can be exercised on API ≥ 34 devices; it applies to Android `record-video` and `stream-video` alike. - `sim-use android record-video`: the Android recording engine moved out of the top-level forwarder into `AndroidBackend`, giving `record-video` the same three-surface layout as every other cross-platform verb (top-level + `ios` + `android`). Flags and behavior are unchanged; `sim-use record-video` with an Android serial routes to the same implementation. - `make e2e-matrix` (`scripts/e2e-matrix.sh`): iOS E2E across the supported host environments — Xcode 26.x / Xcode 27.x × Device Hub closed at boot (the classic Simulator.app workflow, legacy `indigo` HID) / Device Hub open at boot (CoreDevice `dtuhid`). One leg runs the full suite (default `x27-hub`, the primary workflow once Xcode 27 ships; `--full |all|none` to change), the rest run the new smoke tier (`test-runner.sh --smoke`: describe-ui, tap, type, scroll); legs whose Xcode is not installed are skipped. The package builds once on the xcode-select toolchain (`build_products/` is toolchain-locked); each leg swaps only the *runtime* Xcode — `SIM_USE_TEST_DEVELOPER_DIR`, which the test harness injects as `DEVELOPER_DIR` into every process the suites spawn — and boots a device whose iOS runtime matches the leg's Xcode generation (runtimes are system-wide, so "newest available" would cross-contaminate). Every leg is double-gated against silently green-running the wrong combination: dtuhidd process-tree checks before and after the suites, plus a `SIM_USE_DEBUG` probe that must report the matching HID transport-selection predicate. Per-leg logs and a combination-evidence file land in `.build/e2e-matrix//`. `test-runner.sh` now also accepts arbitrary repeatable `swift test --filter` patterns (previously one suite from a fixed whitelist) and resolves its default simulator by exact device name on the newest *available* runtime (previously a substring match that could pick an unavailable device or an "… Pro Max" sibling). - `swipe` / `touch --coordinate-space native|ui` (top-level and `ios` subcommands): explicit coordinates keep their device-native portrait default, and `ui` opts them into the visual space `describe-ui` prints — endpoints are orientation-calibrated per command (per batch for batch steps), so outline coordinates stay correct on a rotated device (#66). `touch` accepts `ui` in the atomic `--down --up` form only: a split touch could straddle a rotation and land its two halves in different spaces. A failed calibration degrades to native dispatch and says so via the `advisory` envelope key. On Android the flag is accepted and ignored — on the top-level commands and the `android swipe` / `android touch` direct commands alike (display coordinates already rotate with the UI). diff --git a/README.md b/README.md index 8cadd40c..f924a443 100644 --- a/README.md +++ b/README.md @@ -166,8 +166,8 @@ For Android, run `sim-use android init --device ` once to install the br All device-scoped commands accept `--device ` (optional when only one simulator is booted). Three command layers: - * **Top-level** — cross-platform verbs: `ui`, `tap`, `swipe`, `type`, `paste`, `button`, `gesture`, `keyboard-state`, `screenshot`, `record-video`, `app-state`. Same flags on iOS and Android. - * **`sim-use ios `** — iOS-only: `key`, `key-combo`, `key-sequence`, `stream-video`, `batch`. + * **Top-level** — cross-platform verbs: `ui`, `tap`, `swipe`, `type`, `paste`, `button`, `gesture`, `keyboard-state`, `screenshot`, `record-video`, `stream-video`, `app-state`. Same flags on iOS and Android. + * **`sim-use ios `** — iOS-only: `key`, `key-combo`, `key-sequence`, `batch`. * **`sim-use android `** — Android-only: `init`, `devices`, `ping`. Run `sim-use --help` or `sim-use --help` for the full flag set. @@ -326,13 +326,18 @@ The output path goes to stdout; progress messages go to stderr. ### Video streaming & recording ```bash -# MJPEG stream (iOS-only — no Android stream-video implementation) -sim-use ios stream-video --device $UDID --fps 10 --format mjpeg > stream.mjpeg +# MJPEG stream (cross-platform) +sim-use stream-video --device $UDID --fps 10 --format mjpeg > stream.mjpeg # Pipe into ffmpeg -sim-use ios stream-video --device $UDID --fps 30 --format ffmpeg | \ +sim-use stream-video --device $UDID --fps 30 --format ffmpeg | \ ffmpeg -f image2pipe -framerate 30 -i - -c:v libx264 -preset ultrafast out.mp4 +# Native H.264 live stream (Android-only): adb screenrecord passthrough — +# variable frame rate, cheap, high quality. Preview it live in ffplay: +sim-use stream-video --device emulator-5554 --format h264 | \ + ffplay -f h264 -probesize 32 -fflags nobuffer - + # Record MP4 directly (cross-platform) sim-use record-video --device $UDID --output recording.mp4 # 30 fps default sim-use record-video --device $UDID --fps 60 --output smooth.mp4 # up to 60 fps diff --git a/Sources/AndroidBackend/Verbs/AndroidCommand.swift b/Sources/AndroidBackend/Verbs/AndroidCommand.swift index e95d4040..d5283bc5 100644 --- a/Sources/AndroidBackend/Verbs/AndroidCommand.swift +++ b/Sources/AndroidBackend/Verbs/AndroidCommand.swift @@ -29,6 +29,7 @@ public struct AndroidCommand: ParsableCommand { AndroidButtonCommand.self, AndroidScreenshotCommand.self, AndroidRecordVideoCommand.self, + AndroidStreamVideoCommand.self, AndroidTypeCommand.self, ] ) diff --git a/Sources/AndroidBackend/Verbs/AndroidRecordVideoCommand.swift b/Sources/AndroidBackend/Verbs/AndroidRecordVideoCommand.swift index 1e19d07a..d33bbe9c 100644 --- a/Sources/AndroidBackend/Verbs/AndroidRecordVideoCommand.swift +++ b/Sources/AndroidBackend/Verbs/AndroidRecordVideoCommand.swift @@ -155,7 +155,7 @@ public struct AndroidRecordVideoCommand: SimUseExecutableCommand { } } - private static func assertAdbDeviceOnline(adb: Adb, serial: String) throws { + static func assertAdbDeviceOnline(adb: Adb, serial: String) throws { let devices: [Adb.Device] do { devices = try adb.devices() @@ -197,7 +197,7 @@ public struct AndroidRecordVideoCommand: SimUseExecutableCommand { let recordingSize = scale < 1.0 ? baseSize.map { scaledSize($0, scale: scale) } : nil let bitrateSize = recordingSize ?? baseSize let bitrate = bitrateSize.map { H264StreamRecorder.estimateBitrate(width: $0.width, height: $0.height, fps: 30, quality: quality) } - let arguments = screenrecordArguments(serial: serial, sdk: sdk, bitrate: bitrate, size: recordingSize) + let arguments = screenrecordArguments(serial: serial, sdk: sdk, bitrate: bitrate, size: recordingSize, timeLimitOverride: screenrecordTimeLimitOverride()) let recorder = try H264PassthroughRecorder(outputURL: outputURL) var recorderFinalized = false @@ -276,14 +276,14 @@ public struct AndroidRecordVideoCommand: SimUseExecutableCommand { } } - private static func detectSDK(adb: Adb, serial: String) -> Int { + static func detectSDK(adb: Adb, serial: String) -> Int { guard let result = try? adb.shell(serial: serial, args: ["getprop", "ro.build.version.sdk"]) else { return 30 } return Int(result.stdout.trimmingCharacters(in: .whitespacesAndNewlines)) ?? 30 } - private static func detectSize(adb: Adb, serial: String) -> (width: Int, height: Int)? { + static func detectSize(adb: Adb, serial: String) -> (width: Int, height: Int)? { guard let result = try? adb.shell(serial: serial, args: ["wm", "size"]) else { return nil } @@ -318,12 +318,29 @@ public struct AndroidRecordVideoCommand: SimUseExecutableCommand { return nil } + /// Debug override for screenrecord's per-invocation time limit + /// (`SIM_USE_SCREENRECORD_TIME_LIMIT`, seconds). API ≥ 34 devices + /// stream unlimited (`--time-limit 0`), so the segment-restart path + /// never fires naturally there — this forces short segments so tests + /// and manual runs can exercise restarts without an API < 34 device. + static func screenrecordTimeLimitOverride( + environment: [String: String] = ProcessInfo.processInfo.environment + ) -> Int? { + guard let raw = environment["SIM_USE_SCREENRECORD_TIME_LIMIT"], + let value = Int(raw), value > 0 else { + return nil + } + return value + } + /// Build the `adb screenrecord` argument vector. `--time-limit 0` /// (unlimited) is only valid on API ≥ 34; older devices hard-cap at 180 s, /// which the segment loop handles by restarting. - static func screenrecordArguments(serial: String, sdk: Int, bitrate: Int?, size: (width: Int, height: Int)?) -> [String] { + static func screenrecordArguments(serial: String, sdk: Int, bitrate: Int?, size: (width: Int, height: Int)?, timeLimitOverride: Int? = nil) -> [String] { var arguments = ["-s", serial, "exec-out", "screenrecord", "--output-format=h264"] - if sdk >= 34 { + if let timeLimitOverride { + arguments.append(contentsOf: ["--time-limit", "\(timeLimitOverride)"]) + } else if sdk >= 34 { arguments.append(contentsOf: ["--time-limit", "0"]) } if let bitrate { @@ -435,7 +452,7 @@ public struct AndroidRecordVideoCommand: SimUseExecutableCommand { /// while the screencap itself is the dominant cost; raising this /// TODO is the cheaper performance lever to reach for first when /// the frame loop becomes the bottleneck. - private static func captureAndroidScreencap(adbPath: String, serial: String) throws -> Data { + static func captureAndroidScreencap(adbPath: String, serial: String) throws -> Data { let process = Process() process.executableURL = URL(fileURLWithPath: adbPath) process.arguments = ["-s", serial, "exec-out", "screencap", "-p"] diff --git a/Sources/AndroidBackend/Verbs/AndroidStreamVideoCommand.swift b/Sources/AndroidBackend/Verbs/AndroidStreamVideoCommand.swift new file mode 100644 index 00000000..cf6bda8c --- /dev/null +++ b/Sources/AndroidBackend/Verbs/AndroidStreamVideoCommand.swift @@ -0,0 +1,411 @@ +// SPDX-License-Identifier: Apache-2.0 +import ArgumentParser +import Foundation +import os +import SimUseCore +import SimUseVideo + +/// `sim-use android stream-video` — stream live video to stdout (#78). +/// +/// Two engines behind one flag surface: +/// +/// * `h264` — native `adb exec-out screenrecord --output-format=h264 -` +/// passthrough: variable frame rate, cheap, high quality. screenrecord's +/// per-invocation time limit is papered over by restarting it and +/// continuing the byte stream (same segment loop as +/// `AndroidRecordVideoCommand`, minus the muxer); each new segment +/// re-emits SPS/PPS, which mainstream decoders (ffplay, ffmpeg) accept +/// mid-stream. +/// * `mjpeg` / `raw` / `ffmpeg` — a `screencap`-per-frame loop +/// (~7–8 FPS ceiling) with byte-identical on-the-wire framing to the +/// iOS `stream-video` formats, so existing consumers work unchanged. +/// +/// `bgra` is intentionally absent: it is raw `FBVideoStream` pixel output +/// and has no Android analog. +public struct AndroidStreamVideoCommand: SimUseExecutableCommand { + public enum OutputFormat: String, ExpressibleByArgument, Codable, Sendable { + case mjpeg + case raw + case ffmpeg + case h264 + } + + /// Summary of a completed stream run. The video bytes are written to + /// stdout inline during `execute()` — they are a side channel, not part + /// of the Result (same posture as `IOSSimStreamVideoCommand`). The JPEG + /// formats count frames; `h264` is a byte-passthrough with no frame + /// notion, so it reports bytes instead. + public struct ExecutionResult: Codable { + public let framesStreamed: UInt64 + public let bytesStreamed: UInt64 + public let durationSeconds: Double + public let format: OutputFormat + + public init(framesStreamed: UInt64, bytesStreamed: UInt64, durationSeconds: Double, format: OutputFormat) { + self.framesStreamed = framesStreamed + self.bytesStreamed = bytesStreamed + self.durationSeconds = durationSeconds + self.format = format + } + } + + public static let configuration = CommandConfiguration( + commandName: "stream-video", + abstract: "Stream live video from the Android device display to stdout" + ) + + @OptionGroup public var device: AndroidDeviceOptions + + @Option(help: "Output format: h264 (native screenrecord passthrough), mjpeg, raw, ffmpeg (screencap JPEG loop). Default: mjpeg") + public var format: OutputFormat = .mjpeg + + @Option(help: "Frames per second for the JPEG formats (1-30, default: 10). Ignored by --format h264 (native variable frame rate).") + public var fps: Int? + + @Option(help: "JPEG quality for the JPEG formats / bitrate factor for h264 (1-100, default: 80)") + public var quality: Int = 80 + + @Option(help: "Scale factor (0.1-1.0, default: 1.0)") + public var scale: Double = 1.0 + + @Flag(name: .customLong("json"), help: "Emit the unified `{ok, data}` summary envelope after the stream ends. Mirrors the cross-platform `stream-video --json` shape.") + public var jsonOutput: Bool = false + + public init() {} + + public mutating func resolveDeferredArguments() throws { + try device.resolve() + } + + public var simulatorUDIDForDaemon: String? { device.resolved } + + /// Streams raw bytes to the caller's stdout for the lifetime of the + /// command — the daemon transport cannot carry that. + public var daemonBypass: Bool { true } + + public func validate() throws { + try VideoRecordingOptions.validateStreaming(fps: fps, quality: quality, scale: scale) + } + + public func format(_ result: ExecutionResult) -> CommandOutput { + guard result.durationSeconds > 0 else { return .empty } + if result.format == .h264 { + guard result.bytesStreamed > 0 else { return .empty } + let line = String( + format: "Streamed %llu bytes in %.1f seconds\n", + result.bytesStreamed, + result.durationSeconds + ) + return CommandOutput(stderr: line) + } + guard result.framesStreamed > 0 else { return .empty } + let avgFPS = Double(result.framesStreamed) / result.durationSeconds + let line = String( + format: "Streamed %llu frames in %.1f seconds (%.1f FPS average)\n", + result.framesStreamed, + result.durationSeconds, + avgFPS + ) + return CommandOutput(stderr: line) + } + + public func execute() async throws -> ExecutionResult { + try await Self.stream( + serial: device.resolved, + format: format, + fps: fps, + quality: quality, + scale: scale + ) + } + + // MARK: - Shared orchestration + + /// Reusable Android streaming entry point: runs until SIGINT/SIGTERM + /// or the consumer closes stdout, then returns the run summary. The + /// top-level cross-platform `StreamVideo` forwards here for Android + /// UDIDs — symmetric to `AndroidRecordVideoCommand.record`. + public static func stream( + serial: String, + format: OutputFormat, + fps: Int?, + quality: Int, + scale: Double + ) async throws -> ExecutionResult { + let adb = Adb() + try AndroidRecordVideoCommand.assertAdbDeviceOnline(adb: adb, serial: serial) + + let cancellationFlag = CancellationFlag() + let signalObserver = SignalObserver(signals: [SIGINT, SIGTERM]) { + cancellationFlag.cancel() + } + defer { signalObserver.invalidate() } + + switch format { + case .h264: + return try await streamH264( + adb: adb, + serial: serial, + fps: fps, + quality: quality, + scale: scale, + cancellationFlag: cancellationFlag + ) + case .mjpeg, .raw, .ffmpeg: + return try await streamJPEGFrames( + adb: adb, + serial: serial, + format: format, + fps: fps ?? 10, + quality: quality, + scale: scale, + cancellationFlag: cancellationFlag + ) + } + } + + // MARK: - h264 passthrough + + /// Native stream: `adb exec-out screenrecord --output-format=h264 -` + /// copied byte-for-byte to stdout. Mirrors the segment-restart loop in + /// `AndroidRecordVideoCommand.recordVideoAndroidStream` — kept separate + /// because the sink (stdout vs muxer), the error taxonomy (no screencap + /// fallback for an explicitly chosen format), and the end-of-stream + /// semantics (consumer hangup is an orderly stop) all differ. + private static func streamH264( + adb: Adb, + serial: String, + fps: Int?, + quality: Int, + scale: Double, + cancellationFlag: CancellationFlag + ) async throws -> ExecutionResult { + if fps != nil { + FileHandle.standardError.write(Data("note: --fps is ignored for --format h264 (screenrecord records at native variable frame rate)\n".utf8)) + } + + let sdk = AndroidRecordVideoCommand.detectSDK(adb: adb, serial: serial) + let baseSize = AndroidRecordVideoCommand.detectSize(adb: adb, serial: serial) + let recordingSize = scale < 1.0 ? baseSize.map { AndroidRecordVideoCommand.scaledSize($0, scale: scale) } : nil + let bitrateSize = recordingSize ?? baseSize + let bitrate = bitrateSize.map { H264StreamRecorder.estimateBitrate(width: $0.width, height: $0.height, fps: 30, quality: quality) } + let arguments = AndroidRecordVideoCommand.screenrecordArguments( + serial: serial, + sdk: sdk, + bitrate: bitrate, + size: recordingSize, + timeLimitOverride: AndroidRecordVideoCommand.screenrecordTimeLimitOverride() + ) + + FileHandle.standardError.write(Data("Streaming Android device \(serial) (h264 Annex B passthrough)...\n".utf8)) + FileHandle.standardError.write(Data("Press Ctrl+C to stop streaming\n".utf8)) + + let sink = StdoutStreamSink() + let startTime = Date() + var firstSegment = true + + segmentLoop: while true { + if Task.isCancelled || cancellationFlag.isCancelled() || sink.isBroken { break } + + let process = AdbStreamingProcess( + adbPath: adb.binaryPath, + arguments: arguments, + onStdout: { data in + if !sink.write(data) { + // Consumer closed its end (ffplay quit, `head` done) + // — stop producing rather than erroring out. + cancellationFlag.cancel() + } + } + ) + do { + try process.start() + } catch { + if firstSegment { + throw CLIError(errorDescription: "screenrecord unavailable (\(error.localizedDescription)). Use --format mjpeg for the screencap-based stream instead.") + } + throw error + } + firstSegment = false + + let segmentStartBytes = process.stdoutByteCount + while process.isRunning { + if Task.isCancelled || cancellationFlag.isCancelled() || sink.isBroken { break } + try? await cancellableSleep(seconds: 0.05, flag: cancellationFlag) + } + + let stopping = Task.isCancelled || cancellationFlag.isCancelled() || sink.isBroken + if stopping { + process.interrupt() + process.waitForExit(timeout: 2) + break + } + + // The process exited on its own — either the time limit was + // reached (restart to continue) or the device stopped feeding. + let exitCode = process.waitForExit(timeout: 2) + let bytesThisSegment = process.stdoutByteCount - segmentStartBytes + if bytesThisSegment == 0 { + let stderrTail = process.collectedStderr.trimmingCharacters(in: .whitespacesAndNewlines) + if sink.bytesWritten == 0 { + let exitDescription = exitCode.map(String.init) ?? "timeout" + throw CLIError(errorDescription: "screenrecord produced no output (exit \(exitDescription)): \(stderrTail). Use --format mjpeg for the screencap-based stream instead.") + } + throw CLIError(errorDescription: "Android device stopped producing frames during streaming (\(stderrTail))") + } + FileHandle.standardError.write(Data("screenrecord segment ended (time limit); restarting stream (~100-300ms gap; the new segment re-emits SPS/PPS)\n".utf8)) + } + + let elapsed = Date().timeIntervalSince(startTime) + return ExecutionResult( + framesStreamed: 0, + bytesStreamed: sink.bytesWritten, + durationSeconds: elapsed, + format: .h264 + ) + } + + // MARK: - screencap JPEG loop + + /// JPEG-frame stream over a `screencap`-per-frame loop, with the same + /// on-the-wire framing as `IOSSimStreamVideoCommand`: + /// `mjpeg` = multipart/x-mixed-replace with `--mjpegstream` boundaries, + /// `raw` = 4-byte big-endian length prefix per frame, + /// `ffmpeg` = bare concatenated frames. + private static func streamJPEGFrames( + adb: Adb, + serial: String, + format: OutputFormat, + fps: Int, + quality: Int, + scale: Double, + cancellationFlag: CancellationFlag + ) async throws -> ExecutionResult { + FileHandle.standardError.write(Data("Starting screencap-based video stream from Android device \(serial)...\n".utf8)) + FileHandle.standardError.write(Data("Format: \(format.rawValue), FPS: \(fps), Quality: \(quality), Scale: \(scale)\n".utf8)) + FileHandle.standardError.write(Data("Press Ctrl+C to stop streaming\n".utf8)) + + let frameInterval = 1.0 / Double(fps) + let mjpegBoundary = "--mjpegstream" + let sink = StdoutStreamSink() + + if format == .mjpeg { + let header = "HTTP/1.1 200 OK\r\nContent-Type: multipart/x-mixed-replace; boundary=\(mjpegBoundary)\r\n\r\n" + sink.write(Data(header.utf8)) + } + + var frameCount: UInt64 = 0 + let startTime = Date() + let adbPath = adb.binaryPath + + while true { + if Task.isCancelled || cancellationFlag.isCancelled() || sink.isBroken { + break + } + + let frameStartTime = Date() + + do { + let frameData = try AndroidRecordVideoCommand.captureAndroidScreencap(adbPath: adbPath, serial: serial) + let processedData = try await VideoFrameUtilities.processJPEGData(frameData, scale: scale, quality: quality) + + switch format { + case .mjpeg: + let frameHeader = "\(mjpegBoundary)\r\nContent-Type: image/jpeg\r\nContent-Length: \(processedData.count)\r\n\r\n" + sink.write(Data(frameHeader.utf8)) + sink.write(processedData) + sink.write(Data("\r\n".utf8)) + case .raw: + var length = UInt32(processedData.count).bigEndian + sink.write(Data(bytes: &length, count: 4)) + sink.write(processedData) + case .ffmpeg: + sink.write(processedData) + case .h264: + break + } + + frameCount += 1 + + if frameCount % UInt64(max(1, fps)) == 0 { + let elapsed = Date().timeIntervalSince(startTime) + if elapsed > 0 { + let actualFPS = Double(frameCount) / elapsed + FileHandle.standardError.write(Data(String(format: "Captured %llu frames (%.1f FPS actual)\n", frameCount, actualFPS).utf8)) + } + } + } catch { + FileHandle.standardError.write(Data("Error capturing frame: \(error.localizedDescription)\n".utf8)) + } + + let elapsed = Date().timeIntervalSince(frameStartTime) + let sleepTime = frameInterval - elapsed + if sleepTime > 0 { + try? await cancellableSleep(seconds: sleepTime, flag: cancellationFlag) + } + } + + if format == .mjpeg && !sink.isBroken { + sink.write(Data("\(mjpegBoundary)--\r\n".utf8)) + } + + let elapsed = Date().timeIntervalSince(startTime) + return ExecutionResult( + framesStreamed: frameCount, + bytesStreamed: sink.bytesWritten, + durationSeconds: elapsed, + format: format + ) + } +} + +/// POSIX-write stdout sink for streaming bytes. +/// +/// `FileHandle.write` raises an uncatchable ObjC exception when the +/// consumer closes its end of the pipe (ffplay quit, `head -c` done) — +/// killing the stream with a crash instead of a summary. This sink +/// ignores SIGPIPE and reports the broken pipe through `isBroken`, so +/// the frame/segment loops can treat consumer hangup as an orderly +/// end-of-stream. Thread-safe: the h264 path writes from the adb reader +/// callback while the command loop polls `isBroken`. +final class StdoutStreamSink: Sendable { + private let state = OSAllocatedUnfairLock(initialState: (bytes: UInt64(0), broken: false)) + + init() { + signal(SIGPIPE, SIG_IGN) + } + + var bytesWritten: UInt64 { state.withLock { $0.bytes } } + var isBroken: Bool { state.withLock { $0.broken } } + + /// Write all of `data` to stdout. Returns false once the pipe is + /// broken; subsequent calls are no-ops. + @discardableResult + func write(_ data: Data) -> Bool { + guard !isBroken else { return false } + let ok = data.withUnsafeBytes { (buffer: UnsafeRawBufferPointer) -> Bool in + guard let base = buffer.baseAddress else { return true } + var offset = 0 + while offset < buffer.count { + let written = Darwin.write(STDOUT_FILENO, base.advanced(by: offset), buffer.count - offset) + if written > 0 { + offset += written + continue + } + if written < 0 && errno == EINTR { + continue + } + return false + } + return true + } + state.withLock { state in + if ok { + state.bytes += UInt64(data.count) + } else { + state.broken = true + } + } + return ok + } +} diff --git a/Sources/SimUse/Commands/StreamVideo.swift b/Sources/SimUse/Commands/StreamVideo.swift new file mode 100644 index 00000000..47642c50 --- /dev/null +++ b/Sources/SimUse/Commands/StreamVideo.swift @@ -0,0 +1,177 @@ +// SPDX-License-Identifier: Apache-2.0 +import ArgumentParser +import Foundation +import SimUseCore +import SimUseVideo +import AndroidBackend +import iOSSimBackend + +/// Top-level cross-platform `stream-video` verb (#78). Owns the flag +/// surface and resolves the target platform, then delegates to: +/// +/// * `IOSSimStreamVideoCommand.execute()` for iOS Simulator UDIDs +/// (screenshot-capture JPEG formats + the raw `bgra` FBVideoStream). +/// * `AndroidStreamVideoCommand.stream()` for adb serials (native +/// `screenrecord` h264 passthrough + screencap JPEG formats). +struct StreamVideo: SimUseExecutableCommand { + /// Union of both backends' formats. `mjpeg` / `raw` / `ffmpeg` are + /// shared; `bgra` is iOS-only (raw FBVideoStream pixels) and `h264` + /// is Android-only for now (iOS H.264 passthrough is a separate + /// follow-up) — the platform mismatch cases fail with a pointer to + /// the right alternative. + enum OutputFormat: String, ExpressibleByArgument, Codable { + case mjpeg + case raw + case ffmpeg + case bgra + case h264 + } + + struct ExecutionResult: Codable { + let framesStreamed: UInt64 + let bytesStreamed: UInt64? + let durationSeconds: Double + let format: String + } + + static let configuration = CommandConfiguration( + commandName: "stream-video", + abstract: "Stream live video from the device display to stdout" + ) + + @OptionGroup var device: DeviceOptions + + @Option(help: "Output format: mjpeg, raw, ffmpeg (both platforms); bgra (iOS-only); h264 (Android-only). Default: mjpeg") + var format: OutputFormat = .mjpeg + + @Option(help: "Frames per second (1-30, default: 10). Ignored by --format h264 (native variable frame rate).") + var fps: Int? + + @Option(help: "JPEG quality (1-100, default: 80)") + var quality: Int = 80 + + @Option(help: "Scale factor (0.1-1.0, default: 1.0)") + var scale: Double = 1.0 + + @OptionGroup var json: JSONOutputOptions + + var jsonOutput: Bool { json.enabled } + + mutating func resolveDeferredArguments() throws { + try device.resolve() + } + + var simulatorUDIDForDaemon: String? { device.resolved } + + var daemonBypass: Bool { true } + + func validate() throws { + try VideoRecordingOptions.validateStreaming(fps: fps, quality: quality, scale: scale) + } + + func format(_ result: ExecutionResult) -> CommandOutput { + guard result.durationSeconds > 0 else { return .empty } + if result.framesStreamed > 0 { + let avgFPS = Double(result.framesStreamed) / result.durationSeconds + let line = String( + format: "Streamed %llu frames in %.1f seconds (%.1f FPS average)\n", + result.framesStreamed, + result.durationSeconds, + avgFPS + ) + return CommandOutput(stderr: line) + } + if let bytes = result.bytesStreamed, bytes > 0 { + let line = String( + format: "Streamed %llu bytes in %.1f seconds\n", + bytes, + result.durationSeconds + ) + return CommandOutput(stderr: line) + } + return .empty + } + + func execute() async throws -> ExecutionResult { + switch PlatformRouter.resolve(udid: device.resolved) { + case .android: + return try await executeAndroid() + case .iOSSim, .none: + return try await executeIOSSim() + } + } + + /// Map the top-level format onto the iOS backend's enum; nil for the + /// Android-only `h264`. + static func iosFormat(for format: OutputFormat) -> IOSSimStreamVideoCommand.OutputFormat? { + switch format { + case .mjpeg: return .mjpeg + case .raw: return .raw + case .ffmpeg: return .ffmpeg + case .bgra: return .bgra + case .h264: return nil + } + } + + /// Map the top-level format onto the Android backend's enum; nil for + /// the iOS-only `bgra`. + static func androidFormat(for format: OutputFormat) -> AndroidStreamVideoCommand.OutputFormat? { + switch format { + case .mjpeg: return .mjpeg + case .raw: return .raw + case .ffmpeg: return .ffmpeg + case .h264: return .h264 + case .bgra: return nil + } + } + + private func executeIOSSim() async throws -> ExecutionResult { + guard Self.iosFormat(for: format) != nil else { + throw CLIError(errorDescription: "--format h264 is Android-only for now (iOS H.264 passthrough is a separate follow-up). Use record-video for an H.264 file, or mjpeg/raw/ffmpeg to stream.") + } + let sub = makeIOSSubcommand() + let result = try await sub.execute() + return ExecutionResult( + framesStreamed: result.framesStreamed, + bytesStreamed: nil, + durationSeconds: result.durationSeconds, + format: result.format.rawValue + ) + } + + /// Construct the backend command and copy every parsed flag across. + /// A missed field stays in ArgumentParser's wrapper-definition state + /// and traps on first read (#42) — pinned by + /// `ForwarderInitializationGuardTests`. + func makeIOSSubcommand() -> IOSSimStreamVideoCommand { + var sub = IOSSimStreamVideoCommand() + // h264 has no iOS mapping and is rejected before this point; the + // fallback value keeps this constructor total for the guard test. + sub.format = Self.iosFormat(for: format) ?? .mjpeg + sub.fps = fps ?? 10 + sub.quality = quality + sub.scale = scale + sub.device = device + sub.json = json + return sub + } + + private func executeAndroid() async throws -> ExecutionResult { + guard let androidFormat = Self.androidFormat(for: format) else { + throw CLIError(errorDescription: "--format bgra is iOS-only (raw FBVideoStream pixel output). Use h264 for a native Android stream, or mjpeg/raw/ffmpeg.") + } + let result = try await AndroidStreamVideoCommand.stream( + serial: device.resolved, + format: androidFormat, + fps: fps, + quality: quality, + scale: scale + ) + return ExecutionResult( + framesStreamed: result.framesStreamed, + bytesStreamed: result.bytesStreamed, + durationSeconds: result.durationSeconds, + format: result.format.rawValue + ) + } +} diff --git a/Sources/SimUse/main.swift b/Sources/SimUse/main.swift index 554450c2..b009d245 100644 --- a/Sources/SimUse/main.swift +++ b/Sources/SimUse/main.swift @@ -32,7 +32,6 @@ private let iOSOnlyVerbRedirects: [String: String] = [ "key": "sim-use ios key", "key-combo": "sim-use ios key-combo", "key-sequence": "sim-use ios key-sequence", - "stream-video": "sim-use ios stream-video", "batch": "sim-use ios batch", ] @@ -51,8 +50,8 @@ enum EntryPoint { Error: `sim-use \(typed)` is not a top-level command. \ Did you mean `\(canonical)`? - Hint: as of 0.5.x, the five iOS-only verbs (key, key-combo, \ - key-sequence, stream-video, batch) live exclusively under \ + Hint: as of 0.5.x, the iOS-only verbs (key, key-combo, \ + key-sequence, batch) live exclusively under \ `sim-use ios ` — the top-level surface only carries \ verbs that work on both iOS and Android. Re-run with the \ `ios` namespace and your existing flags should keep working: @@ -90,16 +89,17 @@ struct SimUse: AsyncParsableCommand { Gesture.self, MultiTouch.self, RecordVideo.self, + StreamVideo.self, Screenshot.self, AppState.self, Viewer.self, // Daemon + spike helpers. Daemon.self, SpikeDaemon.self, - // Platform-specific namespaces. The five iOS-only HID verbs - // (key, key-combo, key-sequence, stream-video, batch) live - // under `IOSSimCommand` only — the top-level surface only - // carries verbs that work on both platforms. + // Platform-specific namespaces. The four iOS-only HID verbs + // (key, key-combo, key-sequence, batch) live under + // `IOSSimCommand` only — the top-level surface only carries + // verbs that work on both platforms. IOSSimCommand.self, AndroidCommand.self, ] diff --git a/Sources/SimUseVideo/VideoRecordingOptions.swift b/Sources/SimUseVideo/VideoRecordingOptions.swift index eda48995..66d45a31 100644 --- a/Sources/SimUseVideo/VideoRecordingOptions.swift +++ b/Sources/SimUseVideo/VideoRecordingOptions.swift @@ -19,4 +19,21 @@ public enum VideoRecordingOptions { throw ValidationError("Scale must be between 0.1 and 1.0") } } + + /// The `stream-video` variant: streaming caps FPS at 30 (screenshot/ + /// screencap capture cannot sustain more, and the h264 passthrough + /// ignores the flag entirely). + public static func validateStreaming(fps: Int?, quality: Int, scale: Double) throws { + if let fps { + guard fps >= 1 && fps <= 30 else { + throw ValidationError("FPS must be between 1 and 30") + } + } + guard quality >= 1 && quality <= 100 else { + throw ValidationError("Quality must be between 1 and 100") + } + guard scale >= 0.1 && scale <= 1.0 else { + throw ValidationError("Scale must be between 0.1 and 1.0") + } + } } diff --git a/Sources/iOSSimBackend/Verbs/IOSSimStreamVideoCommand.swift b/Sources/iOSSimBackend/Verbs/IOSSimStreamVideoCommand.swift index 43109090..59ed6fb2 100644 --- a/Sources/iOSSimBackend/Verbs/IOSSimStreamVideoCommand.swift +++ b/Sources/iOSSimBackend/Verbs/IOSSimStreamVideoCommand.swift @@ -7,11 +7,10 @@ import FBSimulatorControl import SimUseCore import SimUseVideo -/// iOS Simulator backend for the `stream-video` verb. iOS-only — no -/// Android peer. The Android path used to fail-fast with a redirect -/// to `record-video`; with path B the entire verb only exists under -/// `sim-use ios stream-video`, so an Android caller never reaches -/// this code path in the first place. +/// iOS Simulator backend for the `stream-video` verb. The top-level +/// cross-platform `StreamVideo` forwards iOS UDIDs here (#78); an +/// Android UDID passed directly to `sim-use ios stream-video` is +/// redirected to the surfaces that serve it. public struct IOSSimStreamVideoCommand: SimUseExecutableCommand { public enum OutputFormat: String, ExpressibleByArgument, Codable, Sendable { case mjpeg @@ -67,7 +66,7 @@ public struct IOSSimStreamVideoCommand: SimUseExecutableCommand { PlatformRouter.looksLikeAndroid(arg) { // CLIError so the message survives our run() catch — see // IOSSimKeyCommand for the rationale. - throw CLIError(errorDescription: "stream-video is iOS-only. On Android, use `sim-use record-video --udid \(arg)` to capture an MP4 instead.") + throw CLIError(errorDescription: "`sim-use ios stream-video` only drives iOS simulators. For Android, use `sim-use stream-video --udid \(arg)` (or `sim-use android stream-video`).") } try device.resolve() } @@ -91,19 +90,7 @@ public struct IOSSimStreamVideoCommand: SimUseExecutableCommand { } public func validate() throws { - try Self.validateOptions(fps: fps, quality: quality, scale: scale) - } - - public static func validateOptions(fps: Int, quality: Int, scale: Double) throws { - guard fps >= 1 && fps <= 30 else { - throw ValidationError("FPS must be between 1 and 30") - } - guard quality >= 1 && quality <= 100 else { - throw ValidationError("Quality must be between 1 and 100") - } - guard scale >= 0.1 && scale <= 1.0 else { - throw ValidationError("Scale must be between 0.1 and 1.0") - } + try VideoRecordingOptions.validateStreaming(fps: fps, quality: quality, scale: scale) } public func execute() async throws -> ExecutionResult { diff --git a/Tests/AndroidBackendTests/AndroidRecordVideoArgumentTests.swift b/Tests/AndroidBackendTests/AndroidRecordVideoArgumentTests.swift index c0bf25e5..8856927d 100644 --- a/Tests/AndroidBackendTests/AndroidRecordVideoArgumentTests.swift +++ b/Tests/AndroidBackendTests/AndroidRecordVideoArgumentTests.swift @@ -63,6 +63,27 @@ struct AndroidRecordVideoArgumentTests { #expect(!args.contains("--size")) } + @Test("screenrecordArguments honors the time-limit override on API 34+") + func timeLimitOverrideAPI34() { + let args = AndroidRecordVideoCommand.screenrecordArguments(serial: "emu-1", sdk: 34, bitrate: nil, size: nil, timeLimitOverride: 5) + #expect(args == ["-s", "emu-1", "exec-out", "screenrecord", "--output-format=h264", "--time-limit", "5", "-"]) + } + + @Test("time-limit override applies below API 34 too") + func timeLimitOverrideAPI33() { + let args = AndroidRecordVideoCommand.screenrecordArguments(serial: "emu-1", sdk: 33, bitrate: nil, size: nil, timeLimitOverride: 5) + #expect(args.contains("--time-limit")) + #expect(args.contains("5")) + } + + @Test("screenrecordTimeLimitOverride parses the debug env var") + func timeLimitOverrideEnvParsing() { + #expect(AndroidRecordVideoCommand.screenrecordTimeLimitOverride(environment: [:]) == nil) + #expect(AndroidRecordVideoCommand.screenrecordTimeLimitOverride(environment: ["SIM_USE_SCREENRECORD_TIME_LIMIT": "3"]) == 3) + #expect(AndroidRecordVideoCommand.screenrecordTimeLimitOverride(environment: ["SIM_USE_SCREENRECORD_TIME_LIMIT": "0"]) == nil) + #expect(AndroidRecordVideoCommand.screenrecordTimeLimitOverride(environment: ["SIM_USE_SCREENRECORD_TIME_LIMIT": "abc"]) == nil) + } + @Test("scaledSize halves dimensions and rounds down to even") func scaledSizeRounding() { let scaled = AndroidRecordVideoCommand.scaledSize((width: 1081, height: 2401), scale: 0.5) diff --git a/Tests/AndroidStreamVideoTests.swift b/Tests/AndroidStreamVideoTests.swift new file mode 100644 index 00000000..aa4bd4c0 --- /dev/null +++ b/Tests/AndroidStreamVideoTests.swift @@ -0,0 +1,145 @@ +// SPDX-License-Identifier: Apache-2.0 +import Foundation +import Testing + +@Suite("Android Stream Video Tests", .serialized, .enabled(if: isAndroidE2EEnabled)) +struct AndroidStreamVideoTests { + @Test("h264 passthrough emits Annex B bytes and stops cleanly on SIGTERM") + func h264Smoke() async throws { + let result = try await streamForDuration(format: "h264", duration: 4.0) + + #expect(isAcceptableStreamExitCode(result.exitCode), "Unexpected exit code: \(result.exitCode)") + #expect(result.stderr.contains("h264 Annex B passthrough")) + #expect(result.stdout.count > 1_000, "expected a real byte stream, got \(result.stdout.count) bytes") + // screenrecord's stream opens with an Annex B start code (SPS). + #expect(result.stdout.starts(with: [0x00, 0x00, 0x00, 0x01])) + } + + @Test("mjpeg stream carries iOS-parity multipart framing") + func mjpegSmoke() async throws { + let result = try await streamForDuration(format: "mjpeg", duration: 4.0) + + #expect(isAcceptableStreamExitCode(result.exitCode), "Unexpected exit code: \(result.exitCode)") + let text = String(decoding: result.stdout.prefix(4096), as: UTF8.self) + #expect(text.contains("--mjpegstream")) + #expect(text.contains("Content-Type: image/jpeg")) + #expect(result.stderr.contains("Format: mjpeg")) + } + + @Test("raw format prefixes each frame with a 4-byte length") + func rawFraming() async throws { + let result = try await streamForDuration(format: "raw", duration: 3.0) + + #expect(isAcceptableStreamExitCode(result.exitCode), "Unexpected exit code: \(result.exitCode)") + try #require(result.stdout.count > 8) + let length = result.stdout.prefix(4).reduce(UInt32(0)) { ($0 << 8) | UInt32($1) } + // The prefix must describe a plausible frame that the stream had + // room to carry (screencap frames are tens of KB to a few MB). + #expect(length > 100) + #expect(Int(length) <= result.stdout.count) + } + + @Test("h264 stream survives a screenrecord segment restart") + func segmentRestart() async throws { + // The override forces 2-second screenrecord segments, so a ~7 s + // stream must cross at least one restart boundary. + let result = try await streamForDuration( + format: "h264", + duration: 7.0, + environment: ["SIM_USE_SCREENRECORD_TIME_LIMIT": "2"] + ) + + #expect(isAcceptableStreamExitCode(result.exitCode), "Unexpected exit code: \(result.exitCode)") + #expect(result.stderr.contains("restarting stream"), "expected a segment restart, stderr: \(result.stderr)") + #expect(result.stdout.count > 1_000) + } + + @Test("consumer hangup ends the stream without error") + func consumerHangup() async throws { + let serial = try AndroidE2E.requireSerial() + let simUsePath = try TestHelpers.getSimUsePath() + + let process = Process() + process.executableURL = URL(fileURLWithPath: simUsePath) + process.arguments = ["android", "stream-video", "--format", "h264", "--device", serial] + + let stdoutPipe = Pipe() + let stderrPipe = Pipe() + process.standardOutput = stdoutPipe + process.standardError = stderrPipe + + try process.run() + + // Let the stream produce some bytes, then hang up the consumer end. + // The unread pipe may already be full by then — the child's blocked + // write must fail over to EPIPE, not wedge. + try await Task.sleep(nanoseconds: 2_000_000_000) + stdoutPipe.fileHandleForReading.closeFile() + + try await TestHelpers.waitForProcessExit( + process, + timeout: 10.0, + description: "stream-video did not exit after the consumer closed stdout" + ) + + let stderrText = String( + decoding: stderrPipe.fileHandleForReading.readDataToEndOfFile(), + as: UTF8.self + ) + // Hangup is an orderly stop: summary printed, exit 0 — not a crash. + #expect(process.terminationStatus == 0, "expected clean exit, got \(process.terminationStatus); stderr: \(stderrText)") + #expect(stderrText.contains("Streamed")) + } + + // MARK: - Helpers + + private func streamForDuration( + format: String, + duration: TimeInterval, + environment: [String: String] = [:] + ) async throws -> (stdout: Data, stderr: String, exitCode: Int32) { + let serial = try AndroidE2E.requireSerial() + let simUsePath = try TestHelpers.getSimUsePath() + + let process = Process() + process.executableURL = URL(fileURLWithPath: simUsePath) + process.arguments = ["android", "stream-video", "--format", format, "--device", serial] + if !environment.isEmpty { + process.environment = ProcessInfo.processInfo.environment.merging(environment) { _, new in new } + } + + let stdoutPipe = Pipe() + let stderrPipe = Pipe() + process.standardOutput = stdoutPipe + process.standardError = stderrPipe + + // Drain stdout continuously so the child never blocks on a full pipe. + let stdoutReadTask = Task { + try stdoutPipe.fileHandleForReading.readToEnd() ?? Data() + } + + try process.run() + try await Task.sleep(nanoseconds: UInt64(duration * 1_000_000_000)) + process.terminate() + + try await TestHelpers.waitForProcessExit( + process, + timeout: 10.0, + description: "stream-video process did not exit after terminate" + ) + + let stdout = (try? await stdoutReadTask.value) ?? Data() + let stderrText = String( + decoding: stderrPipe.fileHandleForReading.readDataToEndOfFile(), + as: UTF8.self + ) + return (stdout: stdout, stderr: stderrText, exitCode: process.terminationStatus) + } + + private func isAcceptableStreamExitCode(_ code: Int32) -> Bool { + // 0 = clean stop, the rest are signal-mediated exits a supervisor + // may surface (SIGTERM/SIGKILL/SIGINT variants). + let acceptable: Set = [0, 9, 15, 130, 137, 143] + return acceptable.contains(code) + } +} diff --git a/Tests/ForwarderInitializationGuardTests.swift b/Tests/ForwarderInitializationGuardTests.swift index d34f692f..29cb246e 100644 --- a/Tests/ForwarderInitializationGuardTests.swift +++ b/Tests/ForwarderInitializationGuardTests.swift @@ -163,6 +163,11 @@ struct ForwarderInitializationGuardTests { assertFullyInitialized(try RecordVideo.parse(["--udid", iosUDID]).makeIOSSubcommand()) } + @Test("stream-video forwarder copies every field") + func streamVideo() throws { + assertFullyInitialized(try StreamVideo.parse(["--udid", iosUDID]).makeIOSSubcommand()) + } + @Test("multi-touch forwarder copies every field") func multiTouch() throws { assertFullyInitialized(try MultiTouch.parse([ diff --git a/Tests/StreamVideoFormatMappingTests.swift b/Tests/StreamVideoFormatMappingTests.swift new file mode 100644 index 00000000..c4225b3e --- /dev/null +++ b/Tests/StreamVideoFormatMappingTests.swift @@ -0,0 +1,50 @@ +// SPDX-License-Identifier: Apache-2.0 +import Testing +@testable import SimUse +import iOSSimBackend +import AndroidBackend + +/// Pins the top-level `stream-video` format union: the shared JPEG +/// formats map onto both backends, while the platform-exclusive ones +/// (`bgra` iOS-only, `h264` Android-only) map to nil so the forwarder +/// fails with a redirect instead of forwarding an impossible format. +@Suite("StreamVideo format mapping") +struct StreamVideoFormatMappingTests { + @Test("shared formats map onto the iOS backend enum") + func iosSharedFormats() { + #expect(StreamVideo.iosFormat(for: .mjpeg) == .mjpeg) + #expect(StreamVideo.iosFormat(for: .raw) == .raw) + #expect(StreamVideo.iosFormat(for: .ffmpeg) == .ffmpeg) + #expect(StreamVideo.iosFormat(for: .bgra) == .bgra) + } + + @Test("h264 has no iOS mapping") + func iosRejectsH264() { + #expect(StreamVideo.iosFormat(for: .h264) == nil) + } + + @Test("shared formats map onto the Android backend enum") + func androidSharedFormats() { + #expect(StreamVideo.androidFormat(for: .mjpeg) == .mjpeg) + #expect(StreamVideo.androidFormat(for: .raw) == .raw) + #expect(StreamVideo.androidFormat(for: .ffmpeg) == .ffmpeg) + #expect(StreamVideo.androidFormat(for: .h264) == .h264) + } + + @Test("bgra has no Android mapping") + func androidRejectsBGRA() { + #expect(StreamVideo.androidFormat(for: .bgra) == nil) + } + + @Test("default format is mjpeg, matching the per-platform subcommands") + func defaultFormat() throws { + let command = try StreamVideo.parse(["--udid", "00000000-0000-0000-0000-000000000000"]) + #expect(command.format == .mjpeg) + } + + @Test("h264 parses at the top level (Android route)") + func h264Parses() throws { + let command = try StreamVideo.parse(["--format", "h264", "--udid", "emulator-5554"]) + #expect(command.format == .h264) + } +} diff --git a/scripts/test-runner-android.sh b/scripts/test-runner-android.sh index 0b229c05..6df9cdde 100755 --- a/scripts/test-runner-android.sh +++ b/scripts/test-runner-android.sh @@ -29,6 +29,7 @@ PLAYGROUND_APK="Playgrounds/Android/app/build/outputs/apk/debug/app-debug.apk" ALL_SUITES=( "AndroidTapTests" + "AndroidStreamVideoTests" "AndroidSwipeScrollTests" "AndroidTypeTests" "AndroidKeyboardStateTests" From f30e0646a2c354c689782c11fa1bcbb0f03996a6 Mon Sep 17 00:00:00 2001 From: onevcat Date: Wed, 29 Jul 2026 11:31:25 +0900 Subject: [PATCH 2/2] =?UTF-8?q?fix:=20address=20stream-video=20review=20?= =?UTF-8?q?=E2=80=94=20reject=20--json,=20guard=20abnormal=20screenrecord?= =?UTF-8?q?=20exits,=20sync=20skill=20docs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three review findings: 1. --json corrupted the stream (P1): the video bytes own stdout, and the generic runner appended the JSON summary envelope to the same stream after execute(). All three stream-video surfaces (top-level, ios, android) now reject --json at validation time with a targeted error; the run summary stays on stderr. Pinned by a parse-level unit test. 2. Blind restart on abnormal screenrecord exit (P2): both segment loops restarted whenever a segment had produced bytes, so a screenrecord that repeatedly died mid-segment (dead encoder/adb) looped forever. Restart now requires a clean exit 0 (the time-limit case); a non-zero exit aborts with screenrecord's stderr — record-video still finalizes and keeps the partial MP4. 3. Bundled skill cheatsheet still said stream-video was iOS-only (P3): the namespace table and the video section now show the cross-platform verb and the Android h264/ffplay one-liner; stale five-verb iOS-only comments in IOSSimCommand and Command+BatchConvertible updated, and IOSSimStreamVideoCommand moved into the cross-platform registration group. Verified: 1222 unit tests green; live AndroidStreamVideoTests suite green post-change (forced segment restart still restarts on exit 0); --json rejection exercised on all three surfaces against a live emulator and simulator. Co-Authored-By: Claude Fable 5 Signed-off-by: onevcat --- CHANGELOG.md | 2 +- .../Verbs/AndroidRecordVideoCommand.swift | 25 +++++++++++++------ .../Verbs/AndroidStreamVideoCommand.swift | 19 +++++++++++--- Sources/SimUse/Commands/StreamVideo.swift | 6 +++++ .../Batch/Command+BatchConvertible.swift | 2 +- .../iOSSimBackend/Verbs/IOSSimCommand.swift | 3 +-- .../Verbs/IOSSimStreamVideoCommand.swift | 10 ++++++-- Tests/StreamVideoFormatMappingTests.swift | 16 ++++++++++++ skills/sim-use/references/cheatsheet.md | 7 +++--- 9 files changed, 70 insertions(+), 20 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bef35ecc..032adf48 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,7 +21,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- Android support for `stream-video`, and a top-level cross-platform `stream-video` verb (#78). `sim-use android stream-video --format h264` streams native `adb screenrecord` H.264 Annex B passthrough to stdout (variable frame rate, survives screenrecord's per-invocation time limit by restarting mid-stream — each new segment re-emits SPS/PPS, which ffplay/ffmpeg accept); `mjpeg` / `raw` / `ffmpeg` provide a screencap-based JPEG loop (~7–8 FPS ceiling) with byte-identical framing to the iOS formats. The new top-level `sim-use stream-video` routes by UDID shape: `bgra` stays iOS-only, `h264` is Android-only for now, and the platform-mismatch cases fail with a pointer to the right alternative. Streaming now also treats consumer hangup (ffplay quit, `head -c` done) as an orderly end-of-stream on Android instead of dying on SIGPIPE. `SIM_USE_SCREENRECORD_TIME_LIMIT=` (debug) forces short screenrecord segments so the restart path can be exercised on API ≥ 34 devices; it applies to Android `record-video` and `stream-video` alike. +- Android support for `stream-video`, and a top-level cross-platform `stream-video` verb (#78). `sim-use android stream-video --format h264` streams native `adb screenrecord` H.264 Annex B passthrough to stdout (variable frame rate, survives screenrecord's per-invocation time limit by restarting mid-stream — each new segment re-emits SPS/PPS, which ffplay/ffmpeg accept); `mjpeg` / `raw` / `ffmpeg` provide a screencap-based JPEG loop (~7–8 FPS ceiling) with byte-identical framing to the iOS formats. The new top-level `sim-use stream-video` routes by UDID shape: `bgra` stays iOS-only, `h264` is Android-only for now, and the platform-mismatch cases fail with a pointer to the right alternative. Streaming now also treats consumer hangup (ffplay quit, `head -c` done) as an orderly end-of-stream on Android instead of dying on SIGPIPE. `SIM_USE_SCREENRECORD_TIME_LIMIT=` (debug) forces short screenrecord segments so the restart path can be exercised on API ≥ 34 devices; it applies to Android `record-video` and `stream-video` alike. `--json` is rejected on every `stream-video` surface — stdout carries the raw video bytes and the envelope would corrupt the stream (the run summary goes to stderr). The Android segment loops only restart screenrecord after a clean exit 0 (time limit); a screenrecord that dies mid-run now aborts with its stderr instead of being blindly restarted — `record-video` still finalizes and keeps the partial MP4. - `sim-use android record-video`: the Android recording engine moved out of the top-level forwarder into `AndroidBackend`, giving `record-video` the same three-surface layout as every other cross-platform verb (top-level + `ios` + `android`). Flags and behavior are unchanged; `sim-use record-video` with an Android serial routes to the same implementation. - `make e2e-matrix` (`scripts/e2e-matrix.sh`): iOS E2E across the supported host environments — Xcode 26.x / Xcode 27.x × Device Hub closed at boot (the classic Simulator.app workflow, legacy `indigo` HID) / Device Hub open at boot (CoreDevice `dtuhid`). One leg runs the full suite (default `x27-hub`, the primary workflow once Xcode 27 ships; `--full |all|none` to change), the rest run the new smoke tier (`test-runner.sh --smoke`: describe-ui, tap, type, scroll); legs whose Xcode is not installed are skipped. The package builds once on the xcode-select toolchain (`build_products/` is toolchain-locked); each leg swaps only the *runtime* Xcode — `SIM_USE_TEST_DEVELOPER_DIR`, which the test harness injects as `DEVELOPER_DIR` into every process the suites spawn — and boots a device whose iOS runtime matches the leg's Xcode generation (runtimes are system-wide, so "newest available" would cross-contaminate). Every leg is double-gated against silently green-running the wrong combination: dtuhidd process-tree checks before and after the suites, plus a `SIM_USE_DEBUG` probe that must report the matching HID transport-selection predicate. Per-leg logs and a combination-evidence file land in `.build/e2e-matrix//`. `test-runner.sh` now also accepts arbitrary repeatable `swift test --filter` patterns (previously one suite from a fixed whitelist) and resolves its default simulator by exact device name on the newest *available* runtime (previously a substring match that could pick an unavailable device or an "… Pro Max" sibling). - `swipe` / `touch --coordinate-space native|ui` (top-level and `ios` subcommands): explicit coordinates keep their device-native portrait default, and `ui` opts them into the visual space `describe-ui` prints — endpoints are orientation-calibrated per command (per batch for batch steps), so outline coordinates stay correct on a rotated device (#66). `touch` accepts `ui` in the atomic `--down --up` form only: a split touch could straddle a rotation and land its two halves in different spaces. A failed calibration degrades to native dispatch and says so via the `advisory` envelope key. On Android the flag is accepted and ignored — on the top-level commands and the `android swipe` / `android touch` direct commands alike (display coordinates already rotate with the UI). diff --git a/Sources/AndroidBackend/Verbs/AndroidRecordVideoCommand.swift b/Sources/AndroidBackend/Verbs/AndroidRecordVideoCommand.swift index d33bbe9c..d83da33c 100644 --- a/Sources/AndroidBackend/Verbs/AndroidRecordVideoCommand.swift +++ b/Sources/AndroidBackend/Verbs/AndroidRecordVideoCommand.swift @@ -210,7 +210,10 @@ public struct AndroidRecordVideoCommand: SimUseExecutableCommand { }) var firstSegment = true - var disconnected = false + // Set when the stream ends abnormally (device stopped feeding or + // screenrecord died); the recorder still finalizes so the partial + // MP4 survives, then the failure surfaces as the thrown error. + var streamFailure: String? segmentLoop: while true { if Task.isCancelled || cancellationFlag.isCancelled() || fatalBox.first != nil { break } @@ -244,18 +247,26 @@ public struct AndroidRecordVideoCommand: SimUseExecutableCommand { break } - // The process exited on its own — either the API-level time limit - // was reached (restart to continue) or the device stopped feeding. + // The process exited on its own — a clean exit 0 is the API-level + // time limit (restart to continue); anything else is a dead + // device, adb, or encoder, which must NOT be blind-restarted + // into a crash loop just because the segment produced bytes. let exitCode = process.waitForExit(timeout: 2) let bytesThisSegment = process.stdoutByteCount - segmentStartBytes + let stderrTail = process.collectedStderr.trimmingCharacters(in: .whitespacesAndNewlines) if bytesThisSegment == 0 { if !pipeline.firstFrameReceived { let exitDescription = exitCode.map(String.init) ?? "timeout" throw ScreenrecordUnavailableError( - underlying: "screenrecord produced no output (exit \(exitDescription)): \(process.collectedStderr.trimmingCharacters(in: .whitespacesAndNewlines))" + underlying: "screenrecord produced no output (exit \(exitDescription)): \(stderrTail)" ) } - disconnected = true + streamFailure = "Android device stopped producing frames during recording" + break segmentLoop + } + guard exitCode == 0 else { + let exitDescription = exitCode.map(String.init) ?? "timeout" + streamFailure = "screenrecord exited unexpectedly (exit \(exitDescription)): \(stderrTail)" break segmentLoop } FileHandle.standardError.write(Data("screenrecord segment ended (Android time limit); restarting (~100-300ms gap)\n".utf8)) @@ -271,8 +282,8 @@ public struct AndroidRecordVideoCommand: SimUseExecutableCommand { } if let fatal = fatalBox.first { throw fatal } - if disconnected { - throw CLIError(errorDescription: "Android device stopped producing frames during recording; partial recording saved to \(outputURL.path)") + if let streamFailure { + throw CLIError(errorDescription: "\(streamFailure); partial recording saved to \(outputURL.path)") } } diff --git a/Sources/AndroidBackend/Verbs/AndroidStreamVideoCommand.swift b/Sources/AndroidBackend/Verbs/AndroidStreamVideoCommand.swift index cf6bda8c..3750fae7 100644 --- a/Sources/AndroidBackend/Verbs/AndroidStreamVideoCommand.swift +++ b/Sources/AndroidBackend/Verbs/AndroidStreamVideoCommand.swift @@ -68,7 +68,7 @@ public struct AndroidStreamVideoCommand: SimUseExecutableCommand { @Option(help: "Scale factor (0.1-1.0, default: 1.0)") public var scale: Double = 1.0 - @Flag(name: .customLong("json"), help: "Emit the unified `{ok, data}` summary envelope after the stream ends. Mirrors the cross-platform `stream-video --json` shape.") + @Flag(name: .customLong("json"), help: "Unavailable on stream-video — the video bytes own stdout. The flag exists only so agents get a targeted error instead of 'Unknown option'.") public var jsonOutput: Bool = false public init() {} @@ -84,6 +84,11 @@ public struct AndroidStreamVideoCommand: SimUseExecutableCommand { public var daemonBypass: Bool { true } public func validate() throws { + // stdout carries the raw video bytes; the JSON envelope would be + // appended to the same stream after execute() and corrupt it. + if jsonOutput { + throw ValidationError("--json is not available on stream-video: stdout carries the raw video bytes and the envelope would corrupt the stream. The run summary is printed to stderr instead.") + } try VideoRecordingOptions.validateStreaming(fps: fps, quality: quality, scale: scale) } @@ -241,18 +246,24 @@ public struct AndroidStreamVideoCommand: SimUseExecutableCommand { break } - // The process exited on its own — either the time limit was - // reached (restart to continue) or the device stopped feeding. + // The process exited on its own — a clean exit 0 is the time + // limit (restart to continue); anything else is a dead device, + // adb, or encoder, which must NOT be blind-restarted into a + // crash loop just because the segment produced some bytes. let exitCode = process.waitForExit(timeout: 2) let bytesThisSegment = process.stdoutByteCount - segmentStartBytes + let stderrTail = process.collectedStderr.trimmingCharacters(in: .whitespacesAndNewlines) if bytesThisSegment == 0 { - let stderrTail = process.collectedStderr.trimmingCharacters(in: .whitespacesAndNewlines) if sink.bytesWritten == 0 { let exitDescription = exitCode.map(String.init) ?? "timeout" throw CLIError(errorDescription: "screenrecord produced no output (exit \(exitDescription)): \(stderrTail). Use --format mjpeg for the screencap-based stream instead.") } throw CLIError(errorDescription: "Android device stopped producing frames during streaming (\(stderrTail))") } + guard exitCode == 0 else { + let exitDescription = exitCode.map(String.init) ?? "timeout" + throw CLIError(errorDescription: "screenrecord exited unexpectedly (exit \(exitDescription)) mid-stream: \(stderrTail)") + } FileHandle.standardError.write(Data("screenrecord segment ended (time limit); restarting stream (~100-300ms gap; the new segment re-emits SPS/PPS)\n".utf8)) } diff --git a/Sources/SimUse/Commands/StreamVideo.swift b/Sources/SimUse/Commands/StreamVideo.swift index 47642c50..002cfb89 100644 --- a/Sources/SimUse/Commands/StreamVideo.swift +++ b/Sources/SimUse/Commands/StreamVideo.swift @@ -66,6 +66,12 @@ struct StreamVideo: SimUseExecutableCommand { var daemonBypass: Bool { true } func validate() throws { + // stdout carries the raw video bytes; the JSON envelope would be + // appended to the same stream after execute() and corrupt it for + // any consumer. + if json.enabled { + throw ValidationError("--json is not available on stream-video: stdout carries the raw video bytes and the envelope would corrupt the stream. The run summary is printed to stderr instead.") + } try VideoRecordingOptions.validateStreaming(fps: fps, quality: quality, scale: scale) } diff --git a/Sources/iOSSimBackend/Batch/Command+BatchConvertible.swift b/Sources/iOSSimBackend/Batch/Command+BatchConvertible.swift index e1b0c416..8b222502 100644 --- a/Sources/iOSSimBackend/Batch/Command+BatchConvertible.swift +++ b/Sources/iOSSimBackend/Batch/Command+BatchConvertible.swift @@ -7,7 +7,7 @@ import SimUseCore /// Conformance attached to the iOS-side verb command structs so the /// `batch` step parser can lift a parsed step into a sequence of /// `BatchPrimitive`s. iOS-only HID verbs (`key`, `key-combo`, -/// `key-sequence`, `stream-video`, `batch`) have no top-level +/// `key-sequence`, `batch`) have no top-level /// cross-platform wrapper, and the cross-platform verbs (Tap / Type / /// Paste / Button / Swipe / Touch / Gesture) keep all their /// batch-relevant state on their `IOSSimCommand` sub-struct — diff --git a/Sources/iOSSimBackend/Verbs/IOSSimCommand.swift b/Sources/iOSSimBackend/Verbs/IOSSimCommand.swift index b8ff2459..4a8b52de 100644 --- a/Sources/iOSSimBackend/Verbs/IOSSimCommand.swift +++ b/Sources/iOSSimBackend/Verbs/IOSSimCommand.swift @@ -19,7 +19,6 @@ import Foundation /// * `sim-use ios key` (HID keycode press) /// * `sim-use ios key-combo` (HID modifier + key) /// * `sim-use ios key-sequence` (HID sequence of keycodes) -/// * `sim-use ios stream-video` (live mjpeg/raw/bgra stream) /// * `sim-use ios batch` (HID-session-pinned step runner) public struct IOSSimCommand: ParsableCommand { public static let configuration = CommandConfiguration( @@ -48,11 +47,11 @@ public struct IOSSimCommand: ParsableCommand { IOSSimGestureCommand.self, IOSSimMultiTouchCommand.self, IOSSimRecordVideoCommand.self, + IOSSimStreamVideoCommand.self, // iOS-only verbs — reachable only here. IOSSimKeyCommand.self, IOSSimKeyComboCommand.self, IOSSimKeySequenceCommand.self, - IOSSimStreamVideoCommand.self, IOSSimBatchCommand.self, ] ) diff --git a/Sources/iOSSimBackend/Verbs/IOSSimStreamVideoCommand.swift b/Sources/iOSSimBackend/Verbs/IOSSimStreamVideoCommand.swift index 59ed6fb2..042c8cf2 100644 --- a/Sources/iOSSimBackend/Verbs/IOSSimStreamVideoCommand.swift +++ b/Sources/iOSSimBackend/Verbs/IOSSimStreamVideoCommand.swift @@ -22,8 +22,9 @@ public struct IOSSimStreamVideoCommand: SimUseExecutableCommand { /// Summary of a completed stream run. The actual video bytes are /// written to stdout inline during `execute()` — they are a side /// channel, not part of the Result. Streaming commands bypass the - /// daemon transport for exactly this reason, but the typed Result - /// still powers a future `--json` flag that emits the summary alone. + /// daemon transport for exactly this reason, and `--json` is + /// rejected in validate(): the envelope would be appended to the + /// same stdout as the video bytes and corrupt the stream. public struct ExecutionResult: Codable { public let framesStreamed: UInt64 public let durationSeconds: Double @@ -90,6 +91,11 @@ public struct IOSSimStreamVideoCommand: SimUseExecutableCommand { } public func validate() throws { + // stdout carries the raw video bytes; the JSON envelope would be + // appended to the same stream after execute() and corrupt it. + if json.enabled { + throw ValidationError("--json is not available on stream-video: stdout carries the raw video bytes and the envelope would corrupt the stream. The run summary is printed to stderr instead.") + } try VideoRecordingOptions.validateStreaming(fps: fps, quality: quality, scale: scale) } diff --git a/Tests/StreamVideoFormatMappingTests.swift b/Tests/StreamVideoFormatMappingTests.swift index c4225b3e..96807a73 100644 --- a/Tests/StreamVideoFormatMappingTests.swift +++ b/Tests/StreamVideoFormatMappingTests.swift @@ -47,4 +47,20 @@ struct StreamVideoFormatMappingTests { let command = try StreamVideo.parse(["--format", "h264", "--udid", "emulator-5554"]) #expect(command.format == .h264) } + + // stdout carries the raw video bytes, so the summary envelope can + // never share it — all three surfaces must reject the flag at + // validation time rather than corrupt the stream after the fact. + @Test("--json is rejected on every stream-video surface") + func jsonRejectedEverywhere() { + #expect(throws: (any Error).self) { + _ = try StreamVideo.parse(["--json", "--udid", "00000000-0000-0000-0000-000000000000"]) + } + #expect(throws: (any Error).self) { + _ = try AndroidStreamVideoCommand.parse(["--json", "--device", "emulator-5554"]) + } + #expect(throws: (any Error).self) { + _ = try IOSSimStreamVideoCommand.parse(["--json", "--udid", "00000000-0000-0000-0000-000000000000"]) + } + } } diff --git a/skills/sim-use/references/cheatsheet.md b/skills/sim-use/references/cheatsheet.md index 7f7f4361..e939b098 100644 --- a/skills/sim-use/references/cheatsheet.md +++ b/skills/sim-use/references/cheatsheet.md @@ -4,8 +4,8 @@ | Namespace | Scope | Examples | |---|---|---| -| `sim-use ` | Cross-platform (iOS + Android) | `ui`, `tap`, `swipe`, `type`, `paste`, `button`, `gesture`, `screenshot`, `record-video`, `app-state` | -| `sim-use ios ` | iOS Simulator only | `key`, `key-combo`, `key-sequence`, `stream-video`, `batch` | +| `sim-use ` | Cross-platform (iOS + Android) | `ui`, `tap`, `swipe`, `type`, `paste`, `button`, `gesture`, `screenshot`, `record-video`, `stream-video`, `app-state` | +| `sim-use ios ` | iOS Simulator only | `key`, `key-combo`, `key-sequence`, `batch` | | `sim-use android ` | Android device only | `init`, `devices`, `ping` | ## Device resolution @@ -124,7 +124,8 @@ sim-use touch -x 150 -y 250 --down --up --delay 1.0 # long press sim-use screenshot --output shot.png sim-use record-video --output recording.mp4 # H.264, 30 fps default; Ctrl+C to stop sim-use record-video --output smooth.mp4 --fps 60 # iOS: constant rate up to 60 fps (Android ignores --fps, native rate) -sim-use ios stream-video --fps 10 --format mjpeg # iOS only +sim-use stream-video --fps 10 --format mjpeg > out.mjpeg # live JPEG stream (both platforms) +sim-use stream-video --format h264 | ffplay -f h264 - # Android only: native H.264 passthrough (VFR) ``` ### Stopping a backgrounded recording