Repository navigation
feat: record-video captures real H.264 streams - #70
Merged
Merged
Conversation
…polling Replace the screenshot-poll-and-re-encode recorder (capped ~8-10 fps) with native H.264 capture muxed straight into MP4 (passthrough, no re-encode): - iOS: FBSimulatorVideoStream eager H.264 at a constant --fps (1-60, default 30). The elementary stream carries no timestamps, so frames are laid out at exactly 1/fps -- the requested rate is honored and playback is smooth (uniform 16.7 ms spacing at 60 fps). - Android: adb screenrecord --output-format=h264 at the device's native variable rate, stitched across the API<34 180 s per-invocation limit. --fps is ignored there; --quality/--scale map to bitrate/size. New shared infra under Sources/iOSSimBackend/Util: AnnexBStreamParser (NAL splitting + access-unit assembly), H264PassthroughRecorder (AVAssetWriter passthrough; CFR for iOS, host-clock VFR for Android), H264MuxingPipeline; plus AdbStreamingProcess for incremental adb stdout. --fps range widened 1-30 -> 1-60 (default 10 -> 30). The legacy screenshot/screencap recorders are retained as automatic fallbacks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: yuta.ooka <yuta.ooka@lycorp.co.jp>
… Sendable Replace every NSLock + `@unchecked Sendable` in the record-video path with `OSAllocatedUnfairLock<State>`-backed plain `Sendable` classes (CancellationFlag, OnceFlag, FirstErrorBox, H264StreamRecorder, H264PassthroughRecorder, H264MuxingPipeline, AdbStreamingProcess). Non-Sendable writer / subprocess state is confined to the lock, so no type needs an unchecked escape hatch and no withLockUnchecked is used. (Mutex would be cleaner still but requires macOS 15; the package targets macOS 14.) Also wait for the stream's first frame before committing to it: a cold FBSimulatorVideoStream that attaches but pushes nothing now falls back to screenshot capture instead of producing an empty recording. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: yuta.ooka <yuta.ooka@lycorp.co.jp>
…ivery guard, partial-file error wording Fix three issues from @onevcat's review: 1. Android: --quality was silently ignored unless --scale < 1.0, because bitrate was only computed from a size that was only detected when scaled. Detect display size unconditionally (still only pass --size to screenrecord when scale < 1.0) so --bit-rate reaches screenrecord at the default scale too. Verified live: `--quality 20` at default scale now yields `--bit-rate 19740672` on the device's screenrecord command line (previously absent). 2. iOS: CFR mode now detects when the stream delivers fewer frames than the requested rate over the actual recording time (cold start / encoder still releasing) and warns on stderr, since the muxed file would otherwise silently play back faster than real time with no signal to the caller. 3. iOS: mid-stream stream errors that surface after the MP4 was already finalized now say "partial recording saved to <path>", matching the Android branch's wording, so a usable recording isn't discarded just because the command exited non-zero. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: yuta.ooka <yuta.ooka@lycorp.co.jp>
…264-stream Resolves the idb migration merged upstream (76639e4d -> 1f6943f8): static XCFrameworks, FBDeviceControl removed, FutureBridge/BridgeQueues deleted in favor of native async/await APIs (CompanionUtilities). Beyond the textual CHANGELOG.md conflict, this required porting IOSSimRecordVideoCommand.swift off the old FBVideoStream + manual FBDataConsumer + Annex-B passthrough-muxing integration (which no longer compiles) onto upstream's new FBSimulator.startRecording(toFile: configuration:) -- a native in-process file recorder that internally drives the same FBSimulatorVideoStream in eager H.264 mode and muxes to .mp4 via its own AVAssetWriter-backed file writer. This is the same passthrough-muxing architecture this command hand-rolled for the PR; it's now upstream's own maintained implementation, so the custom consumer/pipeline wiring for iOS is gone (Android's adb screenrecord path is untouched and still needs the Annex-B parser/muxer, since idb doesn't drive Android at all). Consequence: H264PassthroughRecorder's CFR (frameRate) mode and its isUnderDelivered under-delivery guard -- added during the earlier PR review to fix onevcat's finding #2 -- are now dead code, since iOS no longer does its own PTS layout (idb's native encoder produces correct timing) and Android is VFR-only. Removed both, along with their unit tests; a PR follow-up comment will explain this to the reviewer. Verified: full unit suite green (849 tests; only the 5 pre-existing unrelated Init-skill failures), e2e RecordVideoTests 6/6 live on a booted simulator (including the short-grace SIGTERM finalize test), and a live Android emulator recording under motion (~60 fps, valid MP4). Repeated rapid back-to-back iOS recordings to probe the cold-start zero-frames issue found during the original PR review -- did not reproduce against the new idb, consistent with upstream's video-layer rewrite. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: yuta.ooka <yuta.ooka@lycorp.co.jp>
Signed-off-by: onevcat <onevcat@gmail.com>
This was referenced Jul 28, 2026
This was referenced Sep 7, 2026
thickfive
pushed a commit
to thickfive/sim-use
that referenced
this pull request
Sep 14, 2026
`stream-video` had no native format on iOS: `mjpeg`/`raw`/`ffmpeg` were a screenshot-per-frame loop and `bgra` was raw pixels, so the only way to watch a simulator live was ~4 fps at ~1.9 MB/s. Meanwhile `record-video` had been driving native H.264 since lycorp-jp#52/lycorp-jp#70 — the capability was already there, just not wired to stdout. Both verbs now build one `FBVideoStreamConfiguration.h264Capture(...)` and differ only in their sink: record-video -> H264MuxingPipeline -> MP4 stream-video -> stdout That muxer's own docs already described it as "shared by the iOS (FBSimulatorVideoStream) and Android (adb screenrecord) capture paths"; iOS simply never connected to it, using idb's separate in-process file writer instead. `AnnexBPipelineConsumer` is the missing adapter from idb's byte-stream consumer protocol to the pipeline. Measured on a booted iPhone 17 Pro (iOS 27.0), 6 s captures: screenshot loop (mjpeg) 24 frames 4.1 fps 11.2 MB screenshot loop (PNG) 26 frames 4.5 fps 92.8 MB native h264 Annex B 144 frames 24.0 fps 1.33 MB The one real trade-off: frame timestamps now come from host arrival time instead of the encoder's sample clock, since Annex B carries no timing. Across three runs each under real touch input, inter-frame jitter measured 6.0-6.8 ms on the new path against 2.7-3.1 ms on the old — both far inside a 33 ms frame at 30 fps, and the `--fps` constant-rate and 100 ms-grace SIGTERM E2E tests both still pass. fmp4 transport was evaluated first and rejected: idb's FBFMP4FrameWriter emits live fragments with no `sidx`/`mfra`, which AVFoundation reads as zero frames, so a recording would have been unplayable in QuickTime and untranscodable to GIF. Annex B keeps the MP4 a regular non-fragmented file. Verified live: both `stream-video --format h264` surfaces plus the top-level route on an iOS UDID, `ffmpeg -f h264` remux, `bgra` non-regression, the GIF path, and the full iOS stream-video and record-video E2E suites. Refs lycorp-jp#132 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Supersedes #52 while preserving its complete commit history.
screenrecordH.264 output directly into MP4 and use idb's native iOS file recorder.Validation
make buildmake test