Skip to content

Commit 8ccb21c

Browse files
onevcatclaude
andcommitted
feat: unify iOS record and stream onto one H.264 capture path
`stream-video` had no native format on iOS: `mjpeg`/`raw`/`ffmpeg` were a screenshot-per-frame loop and `bgra` was raw pixels, so the only way to watch a simulator live was ~4 fps at ~1.9 MB/s. Meanwhile `record-video` had been driving native H.264 since lycorp-jp#52/lycorp-jp#70 — the capability was already there, just not wired to stdout. Both verbs now build one `FBVideoStreamConfiguration.h264Capture(...)` and differ only in their sink: record-video -> H264MuxingPipeline -> MP4 stream-video -> stdout That muxer's own docs already described it as "shared by the iOS (FBSimulatorVideoStream) and Android (adb screenrecord) capture paths"; iOS simply never connected to it, using idb's separate in-process file writer instead. `AnnexBPipelineConsumer` is the missing adapter from idb's byte-stream consumer protocol to the pipeline. Measured on a booted iPhone 17 Pro (iOS 27.0), 6 s captures: screenshot loop (mjpeg) 24 frames 4.1 fps 11.2 MB screenshot loop (PNG) 26 frames 4.5 fps 92.8 MB native h264 Annex B 144 frames 24.0 fps 1.33 MB The one real trade-off: frame timestamps now come from host arrival time instead of the encoder's sample clock, since Annex B carries no timing. Across three runs each under real touch input, inter-frame jitter measured 6.0-6.8 ms on the new path against 2.7-3.1 ms on the old — both far inside a 33 ms frame at 30 fps, and the `--fps` constant-rate and 100 ms-grace SIGTERM E2E tests both still pass. fmp4 transport was evaluated first and rejected: idb's FBFMP4FrameWriter emits live fragments with no `sidx`/`mfra`, which AVFoundation reads as zero frames, so a recording would have been unplayable in QuickTime and untranscodable to GIF. Annex B keeps the MP4 a regular non-fragmented file. Verified live: both `stream-video --format h264` surfaces plus the top-level route on an iOS UDID, `ffmpeg -f h264` remux, `bgra` non-regression, the GIF path, and the full iOS stream-video and record-video E2E suites. Refs lycorp-jp#132 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
1 parent d12bcdd commit 8ccb21c

9 files changed

Lines changed: 205 additions & 82 deletions

‎CHANGELOG.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
77

88
## [Unreleased]
99

10+
### Added
11+
12+
- `stream-video --format h264` now works on iOS, not just Android: a native `FBVideoStream` H.264 Annex B passthrough copied straight to stdout with no host-side codec pass. On a booted iPhone 17 Pro this delivers ~24 fps at ~1.3 MB per 6 s, against the screenshot loop's ~4.1 fps at ~11 MB — roughly 5.9x the frame rate for an eighth of the bytes. It consumes exactly like the Android leg: `sim-use stream-video --format h264 --device $UDID | ffplay -f h264 -probesize 32 -fflags nobuffer -`. The top-level `stream-video` no longer rejects `h264` for iOS targets, so the only platform-exclusive format left is the experimental iOS-only `bgra`. (#132)
13+
14+
### Changed
15+
16+
- iOS `record-video` and `stream-video` now drive the same capture. Both build one `FBVideoStreamConfiguration.h264Capture(fps:quality:scale:)` and differ only in where the encoded bytes land — `record-video` into the shared `H264MuxingPipeline` → MP4 (the muxer Android has always used, and which was written for both), `stream-video` into stdout. Recording no longer goes through idb's separate in-process file writer. Output is equivalent: the MP4 stays a regular non-fragmented file that QuickTime, Finder and `GIFTranscoder` all read, `--fps` is still honoured as a constant rate, and the finalize path still survives a 100 ms-grace SIGTERM across repeated runs. Frame timestamps now come from host arrival time rather than the encoder's sample clock, which measures as ~6 ms of inter-frame jitter against the previous ~3 ms — both far inside a 33 ms frame at 30 fps. (#132)
17+
18+
1019
### Fixed
1120

1221
- The bundled skill preflight now rejects an older `sim-use` CLI before device discovery and prints the Homebrew upgrade command, instead of misdiagnosing newly documented device types as disconnected.

‎README.md‎

Lines changed: 17 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -307,18 +307,22 @@ The output path goes to stdout; progress messages go to stderr.
307307
### Video streaming & recording
308308

309309
```bash
310-
# MJPEG stream (cross-platform)
311-
sim-use stream-video --device $UDID --fps 10 --format mjpeg > stream.mjpeg
310+
# Native H.264 live stream (cross-platform) — the fastest path and the one to
311+
# reach for. iOS drives FBVideoStream at a constant --fps; Android is an adb
312+
# screenrecord passthrough at the device's native variable frame rate.
313+
# Neither pays a host-side codec pass. Preview it live in ffplay:
314+
sim-use stream-video --device $UDID --format h264 | \
315+
ffplay -f h264 -probesize 32 -fflags nobuffer -
312316

313-
# Pipe into ffmpeg
317+
# Or mux it straight to a file
318+
sim-use stream-video --device $UDID --format h264 | ffmpeg -f h264 -i - -c copy out.mp4
319+
320+
# Screenshot-backed formats (deprecated — an order of magnitude slower and
321+
# larger than h264; use h264 unless you specifically need per-frame images)
322+
sim-use stream-video --device $UDID --fps 10 --format mjpeg > stream.mjpeg
314323
sim-use stream-video --device $UDID --fps 30 --format ffmpeg | \
315324
ffmpeg -f image2pipe -framerate 30 -i - -c:v libx264 -preset ultrafast out.mp4
316325

317-
# Native H.264 live stream (Android-only): adb screenrecord passthrough —
318-
# variable frame rate, cheap, high quality. Preview it live in ffplay:
319-
sim-use stream-video --device emulator-5554 --format h264 | \
320-
ffplay -f h264 -probesize 32 -fflags nobuffer -
321-
322326
# Record MP4 directly (cross-platform)
323327
sim-use record-video --device $UDID --output recording.mp4 # 30 fps default
324328
sim-use record-video --device $UDID --fps 60 --output smooth.mp4 # up to 60 fps
@@ -330,6 +334,11 @@ sim-use record-video --device $UDID --format gif # sim-use-
330334
sim-use record-video --device $UDID --format gif --fps 15 --scale 0.4 --output demo.gif
331335
```
332336

337+
`record-video` and `stream-video --format h264` drive the *same* capture on
338+
iOS — one H.264 configuration, differing only in whether the encoded bytes are
339+
muxed into an MP4 or copied to stdout. Measured on a booted iPhone 17 Pro,
340+
`h264` streams ~24 fps at ~220 KB/s where `mjpeg` manages ~4 fps at ~1.9 MB/s.
341+
333342
`record-video` captures a real H.264 stream and muxes it straight into the
334343
MP4 (passthrough — no per-frame screenshot re-encoding). iOS records at a
335344
constant `--fps` (default 30, max 60); Android records at the device's

‎Sources/SimUse/Commands/StreamVideo.swift‎

Lines changed: 13 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -10,15 +10,15 @@ import iOSSimBackend
1010
/// surface and resolves the target platform, then delegates to:
1111
///
1212
/// * `IOSSimStreamVideoCommand.execute()` for iOS Simulator UDIDs
13-
/// (screenshot-capture JPEG formats + the raw `bgra` FBVideoStream).
13+
/// (native `FBVideoStream` h264 passthrough and raw `bgra`, plus the
14+
/// deprecated screenshot-capture formats).
1415
/// * `AndroidStreamVideoCommand.stream()` for adb serials (native
1516
/// `screenrecord` h264 passthrough + screencap JPEG formats).
1617
struct StreamVideo: SimUseExecutableCommand {
17-
/// Union of both backends' formats. `mjpeg` / `raw` / `ffmpeg` are
18-
/// shared; `bgra` is iOS-only (raw FBVideoStream pixels) and `h264`
19-
/// is Android-only for now (iOS H.264 passthrough is a separate
20-
/// follow-up) — the platform mismatch cases fail with a pointer to
21-
/// the right alternative.
18+
/// Union of both backends' formats. `h264` and the deprecated
19+
/// `mjpeg` / `raw` / `ffmpeg` are shared; `bgra` is iOS-only (raw
20+
/// FBVideoStream pixels) and fails on Android with a pointer to the
21+
/// right alternative.
2222
enum OutputFormat: String, ExpressibleByArgument, Codable {
2323
case mjpeg
2424
case raw
@@ -41,10 +41,10 @@ struct StreamVideo: SimUseExecutableCommand {
4141

4242
@OptionGroup var device: DeviceOptions
4343

44-
@Option(help: "Output format: mjpeg, raw, ffmpeg (both platforms); bgra (iOS-only); h264 (Android-only). Default: mjpeg")
44+
@Option(help: "Output format: h264 (native passthrough on both platforms — fastest, recommended); mjpeg, raw, ffmpeg (screenshot-backed, deprecated); bgra (iOS-only raw pixels). Default: mjpeg")
4545
var format: OutputFormat = .mjpeg
4646

47-
@Option(help: "Frames per second (1-30, default: 10). Ignored by --format h264 (native variable frame rate).")
47+
@Option(help: "Frames per second (1-30, default: 10). On iOS --format h264 records at this constant rate; Android's h264 ignores it (native variable frame rate).")
4848
var fps: Int?
4949

5050
@Option(help: "JPEG quality (1-100, default: 80)")
@@ -113,15 +113,15 @@ struct StreamVideo: SimUseExecutableCommand {
113113
}
114114
}
115115

116-
/// Map the top-level format onto the iOS backend's enum; nil for the
117-
/// Android-only `h264`.
116+
/// Map the top-level format onto the iOS backend's enum. Every format
117+
/// has an iOS mapping; only `bgra` lacks an Android one.
118118
static func iosFormat(for format: OutputFormat) -> IOSSimStreamVideoCommand.OutputFormat? {
119119
switch format {
120120
case .mjpeg: return .mjpeg
121121
case .raw: return .raw
122122
case .ffmpeg: return .ffmpeg
123123
case .bgra: return .bgra
124-
case .h264: return nil
124+
case .h264: return .h264
125125
}
126126
}
127127

@@ -138,9 +138,6 @@ struct StreamVideo: SimUseExecutableCommand {
138138
}
139139

140140
private func executeIOSSim() async throws -> ExecutionResult {
141-
guard Self.iosFormat(for: format) != nil else {
142-
throw CLIError(errorDescription: "--format h264 is Android-only for now (iOS H.264 passthrough is a separate follow-up). Use record-video for an H.264 file, or mjpeg/raw/ffmpeg to stream.")
143-
}
144141
let sub = makeIOSSubcommand()
145142
let result = try await sub.execute()
146143
return ExecutionResult(
@@ -157,8 +154,8 @@ struct StreamVideo: SimUseExecutableCommand {
157154
/// `ForwarderInitializationGuardTests`.
158155
func makeIOSSubcommand() -> IOSSimStreamVideoCommand {
159156
var sub = IOSSimStreamVideoCommand()
160-
// h264 has no iOS mapping and is rejected before this point; the
161-
// fallback value keeps this constructor total for the guard test.
157+
// Every top-level format maps to iOS; the fallback only keeps this
158+
// constructor total.
162159
sub.format = Self.iosFormat(for: format) ?? .mjpeg
163160
sub.fps = fps ?? 10
164161
sub.quality = quality
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
// SPDX-License-Identifier: Apache-2.0
2+
import Foundation
3+
@preconcurrency import FBControlCore
4+
import SimUseVideo
5+
6+
/// Bridges idb's byte-stream consumer protocol to the shared Annex B → MP4
7+
/// muxer, letting an `FBSimulatorVideoStream` drive `H264MuxingPipeline`
8+
/// exactly as `adb screenrecord`'s stdout already does on Android — which is
9+
/// the arrangement that pipeline was written for.
10+
///
11+
/// idb delivers encoded bytes on its own capture queue. `ingest` parses and
12+
/// appends synchronously under the pipeline's lock, so the producer sees the
13+
/// same natural backpressure the Android path relies on.
14+
final class AnnexBPipelineConsumer: NSObject, FBDataConsumer {
15+
private let pipeline: H264MuxingPipeline
16+
17+
init(pipeline: H264MuxingPipeline) {
18+
self.pipeline = pipeline
19+
super.init()
20+
}
21+
22+
func consumeData(_ data: Data) {
23+
pipeline.ingest(data)
24+
}
25+
26+
func consumeEndOfFile() {
27+
pipeline.finishIngest()
28+
}
29+
}
Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
// SPDX-License-Identifier: Apache-2.0
2+
import Foundation
3+
@preconcurrency import FBControlCore
4+
5+
extension FBVideoStreamConfiguration {
6+
/// The H.264 capture configuration shared by `record-video` and
7+
/// `stream-video`. Both verbs drive the identical framebuffer encode and
8+
/// differ only in where the encoded bytes land — a muxed MP4 file or
9+
/// stdout — so the configuration lives in one place rather than being
10+
/// spelled out twice and drifting.
11+
static func h264Capture(fps: Int, quality: Int, scale: Double) -> FBVideoStreamConfiguration {
12+
FBVideoStreamConfiguration(
13+
format: .compressedVideo(withCodec: .h264, transport: .annexB),
14+
// Must be non-nil. With no frame rate idb selects a lazy cadence
15+
// that pushes a frame only when the framebuffer reports damage,
16+
// so a still screen would deliver nothing at all.
17+
framesPerSecond: fps,
18+
rateControl: .quality(Double(quality) / 100.0),
19+
scaleFactor: scale,
20+
// A 2 s keyframe interval bounds how long a consumer that joins
21+
// mid-stream waits for its first decodable picture.
22+
keyFrameRate: 2.0
23+
)
24+
}
25+
}

‎Sources/iOSSimBackend/Verbs/IOSSimRecordVideoCommand.swift‎

Lines changed: 29 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -167,38 +167,51 @@ public struct IOSSimRecordVideoCommand: SimUseExecutableCommand {
167167
scale: Double,
168168
cancellationFlag: CancellationFlag
169169
) async throws {
170-
let config = FBVideoStreamConfiguration(
171-
format: .compressedVideo(withCodec: .h264, transport: .annexB),
172-
framesPerSecond: fps,
173-
rateControl: .quality(Double(quality) / 100.0),
174-
scaleFactor: scale,
175-
keyFrameRate: 2.0
176-
)
170+
let config = FBVideoStreamConfiguration.h264Capture(fps: fps, quality: quality, scale: scale)
171+
172+
let recorder = try H264PassthroughRecorder(outputURL: outputURL)
173+
var recorderFinalized = false
174+
defer { if !recorderFinalized { recorder.invalidate() } }
175+
176+
let fatalBox = FirstErrorBox()
177+
let pipeline = H264MuxingPipeline(recorder: recorder, onFatalError: { error in
178+
fatalBox.set(error)
179+
cancellationFlag.cancel()
180+
})
177181

178-
let recording: any FBVideoRecording
182+
let stream: any FBVideoStream
179183
do {
180-
recording = try await simulator.startRecording(toFile: outputURL.path, configuration: config)
184+
stream = try await simulator.createStream(
185+
configuration: config,
186+
to: AnnexBPipelineConsumer(pipeline: pipeline)
187+
)
181188
} catch {
182189
throw RecordingUnavailableError(underlying: error.localizedDescription)
183190
}
184191

185-
while !(Task.isCancelled || cancellationFlag.isCancelled()) {
192+
while !(Task.isCancelled || cancellationFlag.isCancelled() || fatalBox.first != nil) {
186193
try? await cancellableSleep(seconds: 0.1, flag: cancellationFlag)
187194
}
188195

196+
// Stopping the stream is best-effort: a stream that already died
197+
// fails here with a secondary error that would mask the fatal one
198+
// the pipeline latched, and the frames it did deliver are already in
199+
// the recorder either way.
200+
try? await stream.stopStreaming()
201+
pipeline.finishIngest()
202+
189203
do {
190-
_ = try await recording.stop()
204+
try await recorder.finish(stopHostTime: ProcessInfo.processInfo.systemUptime)
205+
recorderFinalized = true
191206
} catch {
192-
// Whether a usable file survived a mid-recording failure is not
193-
// knowable from here (idb owns finalization internally) — check
194-
// the filesystem directly rather than guessing, matching the
195-
// Android branch's "partial recording saved" wording so the
196-
// caller doesn't discard a usable recording on faith alone.
207+
if let fatal = fatalBox.first { throw fatal }
197208
if FileManager.default.fileExists(atPath: outputURL.path) {
198209
throw CLIError(errorDescription: "\(error.localizedDescription); partial recording saved to \(outputURL.path)")
199210
}
200211
throw error
201212
}
213+
214+
if let fatal = fatalBox.first { throw fatal }
202215
}
203216

204217
// MARK: - Screenshot fallback

0 commit comments

Comments
 (0)