Skip to content
9 changes: 6 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,21 +9,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- `stream-video --format h264` now works on iOS, not just Android: a native `FBVideoStream` H.264 stream, carried in MPEG-TS, copied straight to stdout with no host-side codec pass. On a booted iPhone 17 Pro this delivers ~24 fps at ~1.3 MB per 6 s, against the screenshot loop's ~4.1 fps at ~11 MB — roughly 5.9x the frame rate for an eighth of the bytes. Preview it with `sim-use stream-video --format h264 --device $UDID | ffplay -f mpegts -analyzeduration 0 -probesize 32768 -i -`. MPEG-TS rather than the Android leg's bare Annex B on purpose: Annex B carries no presentation timestamps, so a player paces on a guessed frame rate (ffprobe reads a bare stream as 25 fps regardless of `--fps`) and a 30 fps capture drifts ~5 frames further behind every second, unbounded — minutes of lag within a few minutes of viewing. MPEG-TS carries PTS/DTS, and the player's queue measured empty across a 60 s run. The top-level `stream-video` no longer rejects `h264` for iOS targets, so the only platform-exclusive format left is the experimental iOS-only `bgra`. (#132)
- `stream-video --format h264` now works on iOS, not just Android: a native `FBVideoStream` H.264 stream, carried in MPEG-TS, copied straight to stdout with no host-side codec pass. On a booted iPhone 17 Pro this delivers ~24 fps at ~1.3 MB per 6 s, against the screenshot loop's ~4.1 fps at ~11 MB — roughly 5.9x the frame rate for an eighth of the bytes. Preview it with `sim-use stream-video --format h264 --device $UDID | ffplay -f mpegts -probesize 32768 -i -`. It streams at a constant `--fps`, default 30 (the screenshot formats keep their default of 10). MPEG-TS rather than the Android leg's bare Annex B on purpose: Annex B carries no presentation timestamps, so a player paces on a guessed frame rate (ffprobe reads a bare stream as 25 fps regardless of `--fps`) and a 30 fps capture drifts ~5 frames further behind every second, unbounded — minutes of lag within a few minutes of viewing. MPEG-TS carries PTS/DTS, and the player's queue measured empty across a 60 s run. The top-level `stream-video` no longer rejects `h264` for iOS targets, so the only platform-exclusive format left is the experimental iOS-only `bgra`. (#132)

### Changed

- Android `stream-video --format h264` is now carried in MPEG-TS instead of the bare Annex B `screenrecord` produces, matching the iOS surface — one `ffplay -f mpegts -analyzeduration 0 -probesize 32768 -i -` works on both platforms. An elementary stream carries no presentation timestamps, so a player guesses the frame rate (ffprobe reads it as 25 fps regardless of the real rate) and, when the device encodes faster, drains slower than sim-use fills: the lag grew without bound instead of settling, measured at 578 KB of queued video after 20 s and 1376 KB after 30 s of interaction, and observed in practice as a picture minutes behind the device. Frames are re-containered, never re-encoded — a new `MPEGTSMuxer` wraps them with a PTS taken from host arrival time (`screenrecord` is genuinely variable-frame-rate, so wall-clock arrival is the honest timeline). The player's queue now measures empty across a 30 s run. Program tables are emitted up front and re-announced every 100 ms, so a consumer attaching mid-stream can start decoding and a closed pipe is noticed even on a still screen; each picture carries its own clock reference, and a picture resuming after an idle stretch declares a transport discontinuity, since the timeline skipped real elapsed time rather than advancing through it; `--format h264` now reports a frame count in its summary because re-containering parses the stream. End-to-end latency measures ~0.6 s on an emulator, essentially all of it `screenrecord`'s own encode delay (0.58 s measured with sim-use out of the path).
- iOS `stream-video --format mjpeg` / `raw` / `ffmpeg` are deprecated and now warn once on stderr, pointing at `--format h264`. On a simulator the screenshot loop has no remaining advantage — `h264` measured 144 frames in 1.33 MB against `mjpeg`'s 24 in 11.2 MB over the same 6 s — and it is available on every booted simulator, so these will be removed in a later release. **Android keeps its equivalents and does not warn**: `adb screenrecord` is unavailable on some devices, and there the `screencap` loop is the only way to stream at all. `bgra` is unaffected (raw unencoded pixels are not something H.264 substitutes for). (#134)

- Android `stream-video --format h264` is now carried in MPEG-TS instead of the bare Annex B `screenrecord` produces, matching the iOS surface — one `ffplay -f mpegts -probesize 32768 -i -` works on both platforms. An elementary stream carries no presentation timestamps, so a player guesses the frame rate (ffprobe reads it as 25 fps regardless of the real rate) and, when the device encodes faster, drains slower than sim-use fills: the lag grew without bound instead of settling, measured at 578 KB of queued video after 20 s and 1376 KB after 30 s of interaction, and observed in practice as a picture minutes behind the device. Frames are re-containered, never re-encoded — a new `MPEGTSMuxer` wraps them with a PTS taken from host arrival time (`screenrecord` is genuinely variable-frame-rate, so wall-clock arrival is the honest timeline). The player's queue now measures empty across a 30 s run. Program tables are emitted up front and re-announced every 100 ms, so a consumer attaching mid-stream can start decoding and a closed pipe is noticed even on a still screen; each picture carries its own clock reference, and a picture resuming after an idle stretch declares a transport discontinuity, since the timeline skipped real elapsed time rather than advancing through it; `--format h264` now reports a frame count in its summary because re-containering parses the stream. End-to-end latency measures ~0.6 s on an emulator, essentially all of it `screenrecord`'s own encode delay (0.58 s measured with sim-use out of the path).

The stream's timeline advances by the real inter-frame gap but never by more than 0.2 s, which deliberately makes it *not* a reproduction of the wall clock. Capture is variable-frame-rate, so a still screen produces no frames; stamping the next picture with its true arrival time would open a hole in the timeline as long as the pause, and a player has to play through that hole before reaching the new picture — seen as a preview running minutes behind after the screen had been idle. Measured after the fix, time for the picture to catch up once interaction resumes: 0.45 s after 10 s idle, 0.51 s after 30 s, 1.20 s after 60 s; previously this tracked the idle duration. Reproducing the wall clock remains `record-video`'s job, and it keeps real arrival times for exactly that reason. (#135)

- `--format h264`'s documented preview command changed to `ffplay -f mpegts -analyzeduration 0 -probesize 32768 -i -`. The previous `-probesize 32 -fflags nobuffer` never opened an Android stream: capture there is variable-frame-rate, so a still screen thins the stream to a few KB/s of program tables, and ffplay's default five-second *media* analysis window then takes unbounded wall-clock time to fill. `-fflags nobuffer` makes it worse by starving the probe. Both flags are harmless on iOS, which is dense enough to open either way, so one command now covers both platforms.
- `--format h264`'s documented preview command is `ffplay -f mpegts -probesize 32768 -i -`. The previous `-probesize 32 -fflags nobuffer` never opened an Android stream: 32 bytes cannot identify the codec, and `-fflags nobuffer` starves the probe. `-probesize 32768` bounds the open by bytes, which is what matters on a variable-frame-rate source whose still screen sends only program tables — ffplay's default budget is measured in seconds of *media* and can take arbitrarily long in wall-clock terms to fill. `-analyzeduration 0` is deliberately not part of the command: zero selects ffmpeg's default analysis window rather than disabling it, and the open measured identically with and without it. A still Android screen produces no pictures at all, so the player opens once the device moves; iOS is constant-frame-rate and opens at once.

- iOS `record-video` and `stream-video` now share one set of H.264 encoder settings, built by `FBVideoStreamConfiguration.h264Capture(fps:quality:scale:transport:)`, so the two verbs cannot drift apart on how the picture is encoded. They still differ downstream in transport and sink — `record-video` into the shared `H264MuxingPipeline` → MP4 (the muxer Android has always used, and which was written for both), `stream-video` into stdout. Recording no longer goes through idb's separate in-process file writer. Output is equivalent: the MP4 stays a regular non-fragmented file that QuickTime, Finder and `GIFTranscoder` all read, `--fps` is still honoured as a constant rate, and the finalize path still survives a 100 ms-grace SIGTERM across repeated runs. Frame timestamps now come from host arrival time rather than the encoder's sample clock, which measures as ~6 ms of inter-frame jitter against the previous ~3 ms — both far inside a 33 ms frame at 30 fps. (#132)


### Fixed

- iOS `stream-video --format h264` / `bgra`: a consumer that stops reading (ffplay paused, a wedged downstream tool) no longer makes Ctrl-C hang. The native stream went through idb's blocking file writer, which sat in `write(2)` on the encoder thread with `stopStreaming()` waiting behind it, so the command could not end short of SIGKILL. It now writes through the same interruptible stdout sink as Android — one that waits for room and checks cancellation between waits, and leaves the descriptor's flags untouched so the terminal (or stderr under `2>&1`) is never switched into non-blocking mode. A consumer that closes the pipe now ends the stream in an orderly way (stop messages, exit 0) instead of the process dying of SIGPIPE.
- The bundled skill preflight now rejects an older `sim-use` CLI before device discovery and prints the Homebrew upgrade command, instead of misdiagnosing newly documented device types as disconnected.

## [0.14.0] - 2026-08-27
Expand Down
42 changes: 24 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -311,15 +311,17 @@ The output path goes to stdout; progress messages go to stderr.
# reach for. H.264 in MPEG-TS on both platforms, with no host-side codec pass.
# TS carries PTS, so players pace off the stream and stay in sync indefinitely.
sim-use stream-video --device $UDID --format h264 | \
ffplay -f mpegts -analyzeduration 0 -probesize 32768 -i -
ffplay -f mpegts -probesize 32768 -i -

# Muxing the preview to a file works, but it is a live capture, not an
# archive: idle stretches are compressed (see below). For a faithful
# recording use `record-video`.
sim-use stream-video --device $UDID --format h264 | ffmpeg -f mpegts -i - -c copy out.mp4

# Screenshot-backed formats (deprecated — an order of magnitude slower and
# larger than h264; use h264 unless you specifically need per-frame images)
# Screenshot-backed formats. DEPRECATED on iOS (h264 is ~6x the frame rate at
# an eighth of the bytes) and slated for removal there. Retained on Android:
# `screenrecord` is unavailable on some devices, and this loop is the only way
# to stream from those.
sim-use stream-video --device $UDID --fps 10 --format mjpeg > stream.mjpeg
sim-use stream-video --device $UDID --fps 30 --format ffmpeg | \
ffmpeg -f image2pipe -framerate 30 -i - -c:v libx264 -preset ultrafast out.mp4
Expand All @@ -340,28 +342,32 @@ H.264 encoder settings (`--fps`, `--quality`, `--scale`, keyframe interval)
through a single factory, so the two verbs cannot drift apart on how the
picture is encoded. They do differ downstream: recording takes Annex B through
the host-side muxer into an MP4, while streaming takes MPEG-TS straight to
stdout. Measured on a booted iPhone 17 Pro, `h264` streams ~24 fps at
~220 KB/s where `mjpeg` manages ~4 fps at ~1.9 MB/s.
stdout. `h264` streams at a constant `--fps` (default 30, which is also the
streaming cap); measured on a booted iPhone 17 Pro it runs ~220 KB/s at
30 fps where `mjpeg` manages ~4 fps at ~1.9 MB/s.

`stream-video` is a *preview*: its timeline advances by the real gap between
pictures but never by more than 0.2 s, so an idle screen does not leave a hole
a player has to sit through before it reaches the next picture. That is the
right trade when the question is "what is on screen now", and it means the
stream is not a faithful record of elapsed time — piping it to a file turns a
60-second pause into 0.2 seconds of output. `record-video` keeps real arrival
times and is what to use for an archive.

Those two ffplay flags are not decoration. `-analyzeduration 0` is required
because Android's capture is variable-frame-rate: while the screen is still,
`screenrecord` emits no frames at all and the stream thins to just its program
tables (a few KB/s). ffplay's default is to analyse five
seconds of *media* before presenting anything, which on a sparse stream can
take arbitrarily long in wall-clock terms — in practice it never starts.
`-probesize 32768` gives it enough bytes to identify the stream while staying
small enough to stay responsive. Do **not** add `-fflags nobuffer` here: it
starves the probe of the data it needs and ffplay never opens the stream.
(Verified on both platforms; the flags are harmless on iOS, which is
constant-frame-rate and dense enough to open either way.)
times and is what to use for an archive. Ctrl-C ends a stream, and so does
quitting the player: a closed pipe is an orderly stop on both platforms.

`-probesize 32768` is not decoration. ffplay identifies the stream by reading
a bounded amount of it, and the default budget is measured in seconds of
*media*. Android's capture is variable-frame-rate: while the screen is still,
`screenrecord` emits no pictures at all and the stream thins to just its
program tables (a few KB/s), so a media-time budget can take arbitrarily long
in wall-clock terms to fill. Bounding the probe by bytes instead lets it
return after a picture or two. Two things do not help: `-analyzeduration 0`
(zero selects ffmpeg's default window rather than disabling it, and the open
measured identically with and without it) and `-fflags nobuffer` (it starves
the probe of the data it needs and ffplay never opens the stream). One
consequence to know about: a still Android screen sends no pictures, so the
window appears only once something on the device moves. iOS streams at a
constant rate and opens at once.

Both platforms carry `h264` in MPEG-TS, and the reason is worth knowing: a
bare H.264 elementary stream has no timestamps at all, so a player has to
Expand Down
Loading
Loading