Repository navigation
Conversation
Both platforms' screenshot-backed stream formats treated the default `--scale 1.0` / `--quality 80` as an unconditional passthrough, while every MJPEG frame header hardcoded `image/jpeg`. At default settings that handed strict MJPEG clients a PNG payload under a JPEG MIME type. Relabelling the bytes is the wrong fix: the capture is PNG only because we asked for PNG. `captureScreenshotData` hardcoded `format: .png`, so producing JPEG meant a PNG encode, a decode, and a re-encode — three codec passes to change a label. Each format now declares the container its consumers can actually decode. `mjpeg` carries JPEG, because ffmpeg's `mpjpeg` demuxer and the IP-camera clients that read `multipart/x-mixed-replace` reject anything else. `raw` and `ffmpeg` carry the capture's lossless PNG untouched: `raw` delimits frames with its own length prefix, and the `ffmpeg` pipeline README documents sniffs the container via `-f image2pipe` (verified against a live stream — 26 PNG frames demuxed and encoded). On iOS, `SimulatorFrameSource` reads the framebuffer as a CGImage and each sink encodes once into the container it needs, so `--quality` stays honoured (idb's `jpegImageData()` passes nil properties and would have silently dropped it). The recorder now gets its CGImage directly, which also removes the PNG round-trip from the `record-video` screenshot fallback — the one capture path with no fps headroom to spare. On Android, `screencap`'s PNG reaches `raw`/`ffmpeg` with no transcode at all, and only `mjpeg` pays an encode. The run banner reports the frame container (`Frames: jpeg q80`) instead of a `--quality` that never applied to the PNG formats, and `--quality`'s help says which formats it affects. Live-verified on a booted iPhone 17 Pro (iOS 27.0): mjpeg frames carry JPEG under a matching header at 455 KB/frame, where the old PNG payload was 3.5 MB. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
|
Closing in favour of a different approach — see #132. The measurements that settled it, all on the same booted iPhone 17 Pro / iOS 27.0 over 6-second captures:
Native H.264 streaming through Given that, Two pieces of this branch are worth keeping and will be carried into the follow-up work where they still apply:
Branch stays at |
Summary
stream-video's screenshot-backed formats emitted PNG payloads under animage/jpegheader at the default settings. Fixes it by having each format declare the container its consumers can decode, and encoding the capture into that container exactly once.Supersedes #94, which diagnosed the bug correctly — thanks @SunsetWan. Its MJPEG frame-parsing test helper is carried over here.
Root cause
Both platforms shared a fast path that treated
--scale 1.0/--quality 80as an unconditional passthrough, while every MJPEG frame header hardcodedimage/jpeg. iOS screenshots and Androidscreencap -pare both PNG, so the default stream was mislabelled PNG.The capture was PNG only because we asked for PNG:
captureScreenshotDatahardcodedformat: .png. Producing JPEG from that meant a PNG encode, a decode, and a re-encode — three codec passes to change a label.What changed
Each format names its container:
--formatmjpeg--quality)mpjpegdemuxer and the IP-camera clients that readmultipart/x-mixed-replacereject any other containerffmpeg-f image2pipe) sniffs the containerrawSimulatorFrameSourcereads the framebuffer as aCGImage; each sink encodes once into its container. This keeps--qualityhonoured — idb'sjpegImageData()passesnilproperties, so reusingtakeScreenshot(format: .jpeg)would have silently dropped it.screencap's PNG now reachesraw/ffmpegwith no transcode at all. Onlymjpegpays an encode, sincescreencaphas no JPEG option.record-video's iOS screenshot fallback gets itsCGImagestraight from the framebuffer, dropping a PNG encode-then-decode per frame on the one capture path with no fps headroom to spare.Frames: jpeg q80) instead of a--qualitythat never applied to the PNG formats.Verification
make build,make test— 1389 tests pass.mjpeg—Content-Type: image/jpeg,Content-Lengthmatches the payload, payload startsFF D8. 455 KB/frame, where the old mislabelled PNG was 3.5 MB.ffmpeg— opens with a PNG frame; fed to the exact pipeline in the README (ffmpeg -f image2pipe), 26 frames demuxed into 1206x2622 H.264.raw— length prefix describes the frame, payload starts89 50 4E 47.mjpegoutput also demuxes throughffmpeg -f mpjpeg(24 frames) once our HTTP status line is stripped — see the follow-up issue below.--qualityis honoured,mimeTypeasserted against the bytes each container actually produces.image/jpeg" to parsing the frame and checking MIME,Content-Lengthand magic bytes — the weak assertion is why this shipped.Android E2E not run: no emulator reachable in this environment.
Follow-ups (not in this PR)
HTTP/1.1 200 OK, whichffmpeg -f mpjpegcannot parse; stripping it makes the stream demux cleanly. Filed separately.stream-videohas no native H.264 format, unlike Android'sscreenrecordpassthrough.record-videoalready drivesFBVideoStreamConfigurationin H.264 mode, so the plumbing exists. Filed separately.