iOS stream-video has no native H.264 format

[onevcat]

Gap

stream-video's format matrix is asymmetric:

	iOS	Android
mjpeg / raw / ffmpeg	screenshot loop	screencap loop
h264	missing	adb screenrecord passthrough
bgra	raw framebuffer pixels	—

So on iOS every practical live-stream format is a per-frame screenshot loop, which tops out around 4–5 FPS on an iPhone 17 Pro at full resolution (measured while verifying #130) and pays a codec pass per frame. Android's h264 leg streams at the device's native variable frame rate with no per-frame encode on our side.

The plumbing already exists

record-video has driven native H.264 on iOS since #52/#70:

let config = FBVideoStreamConfiguration(
    format: .compressedVideo(withCodec: .h264, transport: .annexB),
    framesPerSecond: fps,
    rateControl: .quality(Double(quality) / 100.0),
    scaleFactor: scale,
    keyFrameRate: 2.0
)

record-video points that at a file. stream-video --format h264 would point the same configuration at stdout, which is what the Android leg already does with screenrecord's Annex B output — and SimUseVideo already has the host-side Annex B parsing from the recording work.

This would also close the "h264 is Android-only for now" caveat in the 0.12.0 changelog and make the stream-video format set symmetric across platforms.

Scope notes

• --fps semantics: iOS record-video honours a constant --fps, so the stream can too (unlike Android's variable rate).
• streamBGRA already streams from FBVideoStream and handles startup/mid-stream failure detection (Surface BGRA stream-video failures as non-zero exit #21); that error taxonomy is the model to follow.
• Worth checking whether ffplay -f h264 -probesize 32 -fflags nobuffer - (the pipeline the README documents for Android) works unchanged against the iOS output.

[onevcat]

Spike results: h264 is the viable native format; .mjpeg is not

Ran the experiment on a booted iPhone 17 Pro / iOS 27.0. idb turns out to expose more than I wrote above — the full format enum is:

// FBControlCore/Configuration/FBVideoStreamConfiguration.swift
public enum FBVideoStreamFormat: Hashable {
  case compressedVideo(withCodec: FBVideoStreamCodec, transport: FBVideoStreamTransport)
  case mjpeg      // raw JPEG bytes, unframed
  case minicap    // 4-byte little-endian length prefix + JPEG
  case bgra
}

.mjpeg and .minicap are fully implemented for simulators (FBSimulatorVideoStream routes both through FBSimulatorVideoStreamFramePusher_VideoToolbox with kCMVideoCodecType_JPEG), and their framing lines up with our formats almost exactly — .mjpeg matches --format ffmpeg, .minicap matches --format raw modulo byte order. So on paper the whole screenshot loop looked replaceable.

It isn't, yet:

path	VideoToolbox codec	result
record-video (native, on main)	H.264	✅ 125 frames in 5 s (~25 fps), 1.04 MB
stream-video --format bgra (on main)	none (Bitmap pusher)	✅ 12.7 MB, clean SIGTERM exit
stream-video spiked to .mjpeg	JPEG	❌ hangs in createStream — 0 bytes, never logs "stream is now running", unresponsive to SIGTERM, needs SIGKILL

The .mjpeg hang reproduces with both framesPerSecond: nil (lazy cadence) and an explicit eager framesPerSecond: fps, so it is not the cadence. Only the format enum differs between the working .bgra run and the hanging one, and .bgra is the one case that never touches VideoToolbox. This project already has history with VideoToolbox session deadlocks (see the GIFTranscoderTests serialization in #101), so a wedged JPEG compression session is the likely culprit.

What this means for scope

1. --format h264 is the win and is unblocked. H.264 through VideoToolbox demonstrably works — record-video depends on it. At ~25 fps versus the screenshot loop's ~4.5 fps on the same device, that's a 5.5x improvement with zero host-side codec passes, and streamBGRA already has the consumer-streaming plumbing (FBFileWriter.syncWriter(withFileDescriptor: STDOUT_FILENO)) plus the startup/mid-stream error taxonomy from Surface BGRA stream-video failures as non-zero exit #21.
2. Native .mjpeg / .minicap need the hang diagnosed first, and should not block the h264 work.
3. The screenshot loop stays either way. H.264 is a temporally-compressed stream: consuming a single frame means decoding and waiting for a keyframe. The per-frame formats (mjpeg/raw/ffmpeg) are what an agent reads when it wants one independently-decodable frame of the current screen, which is sim-use's primary use case. They are a different capability, not a slower version of the same one.