Repository navigation
Deprecate stream-video's screenshot-backed formats (mjpeg / raw / ffmpeg) #134
Description
Follow-up to #133, which added native h264 to iOS stream-video. With that landed, the screenshot-backed formats have no remaining case where they are the right choice for streaming.
Why they go
Measured on a booted iPhone 17 Pro / iOS 27.0, 6-second captures:
--format |
frames | effective fps | bytes |
|---|---|---|---|
mjpeg |
24 | 4.1 | 11.2 MB |
raw / ffmpeg (PNG) |
26 | 4.5 | 92.8 MB |
h264 |
144 | 24.0 | 1.33 MB |
They also have no first-party consumer. skills/sim-use/SKILL.md never mentions stream-video at all — its 10 video references are all screenshot and record-video — and the Viewer polls /api/snapshot for single frames rather than streaming. The "MJPEG lets a browser preview with a zero-JS <img src>" argument doesn't survive contact with the actual shape of the tool either: sim-use writes a byte stream to stdout, not an HTTP response, so that use already requires standing up a server, at which point <video> + MSE is no harder.
They additionally ship two latent bugs, neither worth fixing on a path that's going away:
- Frames don't match their label. At the default
--scale 1.0/--quality 80the frame processor passes the capture through untouched while every MJPEG frame header hardcodesimage/jpeg— but iOS screenshots and Androidscreencap -pare both PNG. Diagnosed in fix: emit JPEG frames for default video streams #94; fix: encode stream-video frames once, into the container they claim #130 fixed it and was closed in favour of this direction. - The MJPEG framing is unparseable by
ffmpeg -f mpjpegbecause of a leadingHTTP/1.1 200 OKstatus line, plus a boundary value that carries its own--against RFC 2046. That's stream-video --format mjpeg is unparseable by ffmpeg's mpjpeg demuxer #131, which this issue supersedes.
Blocked on: Android needs a stream fallback first
This is the real work, and why it wasn't folded into #133.
record-video already falls back to the screencap loop when adb screenrecord is unavailable (old API levels, some devices, rotation mid-capture). stream-video does not. So today, removing mjpeg/raw/ffmpeg from the Android CLI surface would leave those devices with no streaming at all, while recording kept working — an asymmetric capability matrix.
Order of work:
- Give Android
stream-videothe same automatic fallbackrecord-videohas: catchScreenrecordUnavailableError, warn on stderr, continue on thescreencaploop. Needs an emulator to verify, including a device wherescreenrecordgenuinely fails. - Mark
mjpeg/raw/ffmpegdeprecated on all three surfaces (top-level,ios,android): note it in--helpand warn once on stderr, pointing ath264. - Remove them in a later release, keeping the Android
screencaploop as an internal fallback only.
Not in scope
bgra stays. It's iOS-only and experimental, but raw unencoded pixels are something H.264 cannot substitute for, and it costs nothing to keep — it now shares streamNative with h264.
Activity
onevcat commentedon Sep 7, 2026
Revising this issue's plan after implementing #136. The "give Android
stream-videoa screencap fallback first" prerequisite doesn't hold, and the scope should be per-platform rather than global.Why the fallback prerequisite doesn't work
The comparison with
record-videolooked sound but breaks down on inspection.record-video's fallback is transparent because the output is a file: whether frames came fromscreenrecordor thescreencaploop, the result is an MP4 and the caller cannot tell.A stream has no such indirection.
--format h264now emits MPEG-TS (#136); thescreencaploop emits multipart MJPEG or length-prefixed PNG. Falling back mid-stream would change the container under a consumer that has already started decoding, and the consumer's own command line (ffplay -f mpegts …) would no longer be right. There is no way to make it transparent:So an automatic fallback would trade a clear error for a silent format change. The current behaviour — fail with
screenrecord unavailable … Use --format mjpeg for the screencap-based stream instead— is actually the right shape.Revised scope: deprecate on iOS, keep on Android
The formats are not equally redundant across platforms.
iOS — genuinely redundant.
h264is a nativeFBVideoStreamcapture at a constant--fpswith no host-side codec pass, available on every booted simulator. Measured 24 fps / 1.33 MB per 6 s againstmjpeg's 4.1 fps / 11.2 MB. There is no simulator state where the screenshot loop is the better choice, somjpeg/raw/ffmpegcan be deprecated and later removed.Android — still load-bearing.
screenrecordis unavailable on some devices, and thescreencaploop is the only way to stream from those. Removing the formats would leave them with no streaming at all while recording kept working. They stay.That asymmetry is a real capability difference, not an inconsistency to paper over, so the deprecation warnings should say different things:
--format h264instead; these will be removed.--format h264; these remain for devices wherescreenrecordis unavailable.Revised order of work
mjpeg/raw/ffmpegon the iOS surface (--helptext + a one-time stderr warning), and on the top-level verb when the target resolves to a simulator.screenrecord-unavailable path rather than a general-purpose choice.#131 (the MJPEG framing being unparseable by
ffmpeg -f mpjpeg) is not superseded after all, since the framing survives on Android. It should be fixed or documented on its own merits.feat: deprecate the screenshot stream formats on iOS, keep them on An…
feat: deprecate the screenshot stream formats on iOS, keep them on An…
feat: deprecate the screenshot stream formats on iOS, keep them on An…
feat: deprecate the screenshot stream formats on iOS, keep them on An…
feat: deprecate the screenshot stream formats on iOS, keep them on An…
feat: deprecate the screenshot stream formats on iOS, keep them on An…