From fbe39c9ab93e847ebb0c95254390355096cc0de1 Mon Sep 17 00:00:00 2001 From: onevcat Date: Mon, 7 Sep 2026 16:48:42 +0900 Subject: [PATCH] fix: encode stream-video frames once, into the container they claim MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both platforms' screenshot-backed stream formats treated the default `--scale 1.0` / `--quality 80` as an unconditional passthrough, while every MJPEG frame header hardcoded `image/jpeg`. At default settings that handed strict MJPEG clients a PNG payload under a JPEG MIME type. Relabelling the bytes is the wrong fix: the capture is PNG only because we asked for PNG. `captureScreenshotData` hardcoded `format: .png`, so producing JPEG meant a PNG encode, a decode, and a re-encode — three codec passes to change a label. Each format now declares the container its consumers can actually decode. `mjpeg` carries JPEG, because ffmpeg's `mpjpeg` demuxer and the IP-camera clients that read `multipart/x-mixed-replace` reject anything else. `raw` and `ffmpeg` carry the capture's lossless PNG untouched: `raw` delimits frames with its own length prefix, and the `ffmpeg` pipeline README documents sniffs the container via `-f image2pipe` (verified against a live stream — 26 PNG frames demuxed and encoded). On iOS, `SimulatorFrameSource` reads the framebuffer as a CGImage and each sink encodes once into the container it needs, so `--quality` stays honoured (idb's `jpegImageData()` passes nil properties and would have silently dropped it). The recorder now gets its CGImage directly, which also removes the PNG round-trip from the `record-video` screenshot fallback — the one capture path with no fps headroom to spare. On Android, `screencap`'s PNG reaches `raw`/`ffmpeg` with no transcode at all, and only `mjpeg` pays an encode. The run banner reports the frame container (`Frames: jpeg q80`) instead of a `--quality` that never applied to the PNG formats, and `--quality`'s help says which formats it affects. Live-verified on a booted iPhone 17 Pro (iOS 27.0): mjpeg frames carry JPEG under a matching header at 455 KB/frame, where the old PNG payload was 3.5 MB. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: onevcat --- CHANGELOG.md | 5 + README.md | 16 +++ .../Verbs/AndroidStreamVideoCommand.swift | 30 +++-- Sources/SimUseVideo/VideoCommandSupport.swift | 107 +++++++++++---- .../Util/VideoFrameUtilities+Simulator.swift | 33 ++++- .../Verbs/IOSSimRecordVideoCommand.swift | 25 ++-- .../Verbs/IOSSimStreamVideoCommand.swift | 41 +++++- Tests/AndroidStreamVideoTests.swift | 16 ++- Tests/MJPEGTestSupport.swift | 51 +++++++ Tests/StreamVideoTests.swift | 21 ++- Tests/VideoFrameProcessingTests.swift | 124 ++++++++++++++---- 11 files changed, 378 insertions(+), 91 deletions(-) create mode 100644 Tests/MJPEGTestSupport.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index e16141c7..314c1e09 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Fixed + +- `stream-video` frames now match the container they are labelled with, and are encoded exactly once. Both platforms' screenshot-backed formats treated the default `--scale 1.0` / `--quality 80` as an unconditional passthrough while every MJPEG frame header claimed `image/jpeg`, so at default settings strict MJPEG clients were handed PNG payloads under a JPEG MIME type. Each format now declares the container its consumers can actually decode — `mjpeg` carries JPEG (ffmpeg's `mpjpeg` demuxer and IP-camera clients reject anything else), while `raw` and `ffmpeg` carry the capture's lossless PNG untouched, since `raw` delimits frames with its own length prefix and the documented `ffmpeg` pipeline sniffs the container via `-f image2pipe`. On iOS the capture is read off the framebuffer as a `CGImage` and encoded once into the format's container, replacing a PNG encode plus a decode plus a re-encode; on Android `screencap`'s PNG now reaches `raw`/`ffmpeg` with no transcode at all, and only `mjpeg` pays an encode. The iOS `record-video` screenshot fallback stopped round-tripping every frame through PNG as well. The run banner now reports the frame container (`Frames: jpeg q80`) in place of a `--quality` that never applied to the PNG formats, and `--quality`'s help says which formats it affects. (#94 — thanks @SunsetWan for the diagnosis and the MJPEG frame-parsing test helper!) + + ### Fixed - The bundled skill preflight now rejects an older `sim-use` CLI before device discovery and prints the Homebrew upgrade command, instead of misdiagnosing newly documented device types as disconnected. diff --git a/README.md b/README.md index 4e766801..e5aa4edf 100644 --- a/README.md +++ b/README.md @@ -330,6 +330,22 @@ sim-use record-video --device $UDID --format gif # sim-use- sim-use record-video --device $UDID --format gif --fps 15 --scale 0.4 --output demo.gif ``` +`stream-video`'s screenshot-backed formats each carry the container their +consumers can actually decode, and the capture is encoded into it exactly +once — no frame is transcoded just to change its label: + +| `--format` | frames | consumer | +|---|---|---| +| `mjpeg` | JPEG (`--quality`, default 80) | browsers, `ffmpeg -f mpjpeg`, MJPEG clients — none of which accept another container | +| `ffmpeg` | PNG, lossless | `ffmpeg -f image2pipe`, which sniffs the container | +| `raw` | PNG, lossless | your own reader; each frame carries a 4-byte big-endian length prefix | +| `bgra` (iOS) / `h264` (Android) | raw pixels / H.264 Annex B | not screenshot-backed | + +`--quality` therefore only bites on `mjpeg`; the PNG formats are lossless, +and trade roughly 7x the bytes per frame for it (measured on a booted +iPhone 17 Pro home screen: 455 KB/frame at `mjpeg`'s default quality vs +3.5 MB/frame as PNG). Reach for `mjpeg` when the stream leaves the machine. + `record-video` captures a real H.264 stream and muxes it straight into the MP4 (passthrough — no per-frame screenshot re-encoding). iOS records at a constant `--fps` (default 30, max 60); Android records at the device's diff --git a/Sources/AndroidBackend/Verbs/AndroidStreamVideoCommand.swift b/Sources/AndroidBackend/Verbs/AndroidStreamVideoCommand.swift index 3750fae7..8ff827b7 100644 --- a/Sources/AndroidBackend/Verbs/AndroidStreamVideoCommand.swift +++ b/Sources/AndroidBackend/Verbs/AndroidStreamVideoCommand.swift @@ -28,6 +28,17 @@ public struct AndroidStreamVideoCommand: SimUseExecutableCommand { case raw case ffmpeg case h264 + + /// The container this format's consumers can decode, or nil for + /// `h264`, which is a byte passthrough. Matches the iOS surface — + /// `stream-video` promises byte-identical framing across platforms. + func frameContainer(quality: Int) -> FrameContainer? { + switch self { + case .mjpeg: .jpeg(quality: quality) + case .raw, .ffmpeg: .png + case .h264: nil + } + } } /// Summary of a completed stream run. The video bytes are written to @@ -62,7 +73,7 @@ public struct AndroidStreamVideoCommand: SimUseExecutableCommand { @Option(help: "Frames per second for the JPEG formats (1-30, default: 10). Ignored by --format h264 (native variable frame rate).") public var fps: Int? - @Option(help: "JPEG quality for the JPEG formats / bitrate factor for h264 (1-100, default: 80)") + @Option(help: "JPEG quality for mjpeg / bitrate factor for h264 (1-100, default: 80). Ignored by raw/ffmpeg, which carry lossless PNG frames.") public var quality: Int = 80 @Option(help: "Scale factor (0.1-1.0, default: 1.0)") @@ -157,12 +168,15 @@ public struct AndroidStreamVideoCommand: SimUseExecutableCommand { cancellationFlag: cancellationFlag ) case .mjpeg, .raw, .ffmpeg: - return try await streamJPEGFrames( + guard let container = format.frameContainer(quality: quality) else { + throw CLIError(errorDescription: "Format \(format.rawValue) does not carry encoded frames") + } + return try await streamFrames( adb: adb, serial: serial, format: format, + container: container, fps: fps ?? 10, - quality: quality, scale: scale, cancellationFlag: cancellationFlag ) @@ -283,17 +297,17 @@ public struct AndroidStreamVideoCommand: SimUseExecutableCommand { /// `mjpeg` = multipart/x-mixed-replace with `--mjpegstream` boundaries, /// `raw` = 4-byte big-endian length prefix per frame, /// `ffmpeg` = bare concatenated frames. - private static func streamJPEGFrames( + private static func streamFrames( adb: Adb, serial: String, format: OutputFormat, + container: FrameContainer, fps: Int, - quality: Int, scale: Double, cancellationFlag: CancellationFlag ) async throws -> ExecutionResult { FileHandle.standardError.write(Data("Starting screencap-based video stream from Android device \(serial)...\n".utf8)) - FileHandle.standardError.write(Data("Format: \(format.rawValue), FPS: \(fps), Quality: \(quality), Scale: \(scale)\n".utf8)) + FileHandle.standardError.write(Data("Format: \(format.rawValue), FPS: \(fps), Frames: \(container), Scale: \(scale)\n".utf8)) FileHandle.standardError.write(Data("Press Ctrl+C to stop streaming\n".utf8)) let frameInterval = 1.0 / Double(fps) @@ -318,11 +332,11 @@ public struct AndroidStreamVideoCommand: SimUseExecutableCommand { do { let frameData = try AndroidRecordVideoCommand.captureAndroidScreencap(adbPath: adbPath, serial: serial) - let processedData = try await VideoFrameUtilities.processJPEGData(frameData, scale: scale, quality: quality) + let processedData = try VideoFrameUtilities.transcodeFrame(frameData, to: container, scale: scale) switch format { case .mjpeg: - let frameHeader = "\(mjpegBoundary)\r\nContent-Type: image/jpeg\r\nContent-Length: \(processedData.count)\r\n\r\n" + let frameHeader = "\(mjpegBoundary)\r\nContent-Type: \(container.mimeType)\r\nContent-Length: \(processedData.count)\r\n\r\n" sink.write(Data(frameHeader.utf8)) sink.write(processedData) sink.write(Data("\r\n".utf8)) diff --git a/Sources/SimUseVideo/VideoCommandSupport.swift b/Sources/SimUseVideo/VideoCommandSupport.swift index 00175bed..374b0dfc 100644 --- a/Sources/SimUseVideo/VideoCommandSupport.swift +++ b/Sources/SimUseVideo/VideoCommandSupport.swift @@ -61,6 +61,61 @@ public struct VideoWriterStallError: Error, LocalizedError, Equatable { } } +/// The wire container a captured frame is carried in. +/// +/// Stream formats differ in what their consumers can actually decode, so +/// each one names its container and the capture path encodes into it +/// exactly once. Nothing re-containers bytes just to change their label. +public enum FrameContainer: Equatable, Sendable, CustomStringConvertible { + /// Lossless. `--format raw` delimits frames itself and ffmpeg's + /// `image2pipe` demuxer sniffs the container, so both carry PNG + /// straight from the capture with no re-encode. + case png + /// Motion JPEG only accepts JPEG: ffmpeg's `mpjpeg` demuxer and the + /// IP-camera clients that read `multipart/x-mixed-replace` reject + /// every other container. + case jpeg(quality: Int) + + public var mimeType: String { + switch self { + case .png: "image/png" + case .jpeg: "image/jpeg" + } + } + + public var description: String { + switch self { + case .png: "png (lossless)" + case .jpeg(let quality): "jpeg q\(quality)" + } + } + + /// Leading bytes identifying an encoded frame as this container. + var magicBytes: [UInt8] { + switch self { + case .png: [0x89, 0x50, 0x4E, 0x47] + case .jpeg: [0xFF, 0xD8] + } + } + + var typeIdentifier: CFString { + switch self { + case .png: "public.png" as CFString + case .jpeg: "public.jpeg" as CFString + } + } + + var destinationProperties: CFDictionary? { + switch self { + case .png: + // Lossless: `--quality` has no meaning here. + nil + case .jpeg(let quality): + [kCGImageDestinationLossyCompressionQuality: Double(quality) / 100.0] as CFDictionary + } + } +} + public struct VideoFrameUtilities { public static func makeCGImage(from data: Data) -> CGImage? { guard let source = CGImageSourceCreateWithData(data as CFData, nil) else { @@ -69,13 +124,29 @@ public struct VideoFrameUtilities { return CGImageSourceCreateImageAtIndex(source, 0, nil) } - public static func processJPEGData(_ data: Data, scale: Double, quality: Int) async throws -> Data { - if scale < 1.0 { - return try await scaleJPEGData(data, scale: scale, quality: quality) - } else if quality != 80 { - return try await reencodeJPEGData(data, quality: quality) + /// Encode a freshly captured frame into `container`, applying `scale` + /// in the same pass. This is the one codec pass a frame from an + /// un-encoded source has to pay. + public static func encodeFrame(_ image: CGImage, as container: FrameContainer, scale: Double) throws -> Data { + let source = scale < 1.0 ? try resize(image, scale: scale) : image + return try encode(source, as: container) + } + + /// Re-container an already-encoded frame, skipping the work when the + /// bytes are already what the sink needs. Only capture paths that + /// cannot choose their container need this — Android's `screencap` + /// emits PNG and nothing else. + public static func transcodeFrame(_ data: Data, to container: FrameContainer, scale: Double) throws -> Data { + // Passthrough is sound only for a lossless container: JPEG bytes + // carry no record of the quality they were encoded at, so + // honouring `--quality` means encoding them ourselves. + if scale >= 1.0, container == .png, data.starts(with: container.magicBytes) { + return data + } + guard let image = makeCGImage(from: data) else { + throw VideoProcessingError.failedToDecodeImage } - return data + return try encodeFrame(image, as: container, scale: scale) } public static func computeDimensions(for image: CGImage, scale: Double) -> (width: Int, height: Int) { @@ -86,21 +157,18 @@ public struct VideoFrameUtilities { return (max(evenWidth, 2), max(evenHeight, 2)) } - private static func encodeJPEG(_ image: CGImage, quality: Int) throws -> Data { + private static func encode(_ image: CGImage, as container: FrameContainer) throws -> Data { guard let data = CFDataCreateMutable(nil, 0), let destination = CGImageDestinationCreateWithData( data, - "public.jpeg" as CFString, + container.typeIdentifier, 1, nil ) else { throw VideoProcessingError.failedToEncodeImage } - let properties = [ - kCGImageDestinationLossyCompressionQuality: Double(quality) / 100.0 - ] as CFDictionary - CGImageDestinationAddImage(destination, image, properties) + CGImageDestinationAddImage(destination, image, container.destinationProperties) guard CGImageDestinationFinalize(destination) else { throw VideoProcessingError.failedToEncodeImage @@ -109,11 +177,7 @@ public struct VideoFrameUtilities { return data as Data } - private static func scaleJPEGData(_ data: Data, scale: Double, quality: Int) async throws -> Data { - guard let image = makeCGImage(from: data) else { - throw VideoProcessingError.failedToDecodeImage - } - + private static func resize(_ image: CGImage, scale: Double) throws -> CGImage { let dimensions = computeDimensions(for: image, scale: scale) guard let context = CGContext( data: nil, @@ -142,16 +206,9 @@ public struct VideoFrameUtilities { throw VideoProcessingError.failedToEncodeImage } - return try encodeJPEG(scaledImage, quality: quality) + return scaledImage } - private static func reencodeJPEGData(_ data: Data, quality: Int) async throws -> Data { - guard let image = makeCGImage(from: data) else { - throw VideoProcessingError.failedToDecodeImage - } - - return try encodeJPEG(image, quality: quality) - } } public final class H264StreamRecorder: Sendable { diff --git a/Sources/iOSSimBackend/Util/VideoFrameUtilities+Simulator.swift b/Sources/iOSSimBackend/Util/VideoFrameUtilities+Simulator.swift index 67c9ef20..1d545d5c 100644 --- a/Sources/iOSSimBackend/Util/VideoFrameUtilities+Simulator.swift +++ b/Sources/iOSSimBackend/Util/VideoFrameUtilities+Simulator.swift @@ -1,10 +1,11 @@ // SPDX-License-Identifier: Apache-2.0 +import CoreGraphics import Foundation import FBSimulatorControl @preconcurrency import FBControlCore import SimUseVideo -// The FB*-tied capture entry point for the shared frame utilities. +// The FB*-tied capture entry points for the shared frame utilities. // Everything else in `VideoFrameUtilities` is platform-neutral and lives // in SimUseVideo; this extension is the one piece that must stay inside // iOSSimBackend's FB* dep cone. @@ -17,3 +18,33 @@ extension VideoFrameUtilities { return data } } + +/// A framebuffer-attached source of un-encoded frames. +/// +/// `takeScreenshot` encodes on the simulator's behalf, which forces every +/// consumer into whatever container it picked — PNG — and makes a stream +/// that wants JPEG pay a decode plus a re-encode per frame. Reading the +/// framebuffer directly hands out the `CGImage` instead, so each sink +/// encodes exactly once into the container it needs, and the H.264 +/// recorder never encodes to an intermediate container at all. +/// +/// Attaching a consumer to the framebuffer costs a surface handshake, so +/// one source is created per capture session and reused for every frame. +public final class SimulatorFrameSource { + private let image: FBSimulatorImage + + public init(simulator: FBSimulator) async throws { + let framebuffer = try await simulator.connectToFramebuffer() + self.image = FBSimulatorImage.image(with: framebuffer, logger: simulator.logger) + } + + /// The framebuffer's current contents. Throws rather than returning + /// nil so a frame loop can log and continue on a transient miss, + /// matching how it already handles capture failures. + public func currentFrame() throws -> CGImage { + guard let frame = image.image() else { + throw VideoProcessingError.emptyScreenshot + } + return frame + } +} diff --git a/Sources/iOSSimBackend/Verbs/IOSSimRecordVideoCommand.swift b/Sources/iOSSimBackend/Verbs/IOSSimRecordVideoCommand.swift index fe9765a5..ae4cd1a7 100644 --- a/Sources/iOSSimBackend/Verbs/IOSSimRecordVideoCommand.swift +++ b/Sources/iOSSimBackend/Verbs/IOSSimRecordVideoCommand.swift @@ -215,10 +215,11 @@ public struct IOSSimRecordVideoCommand: SimUseExecutableCommand { scale: Double, cancellationFlag: CancellationFlag ) async throws { - let initialFrameData = try await VideoFrameUtilities.captureScreenshotData(from: simulator) - guard let initialImage = VideoFrameUtilities.makeCGImage(from: initialFrameData) else { - throw CLIError(errorDescription: "Failed to decode simulator screenshot") - } + // Frames come off the framebuffer un-encoded: the recorder wants a + // CGImage, so encoding to PNG only to decode it straight back was + // pure overhead on the one path that already has no fps headroom. + let frameSource = try await SimulatorFrameSource(simulator: simulator) + let initialImage = try frameSource.currentFrame() let dimensions = VideoFrameUtilities.computeDimensions(for: initialImage, scale: scale) let recorder = try H264StreamRecorder( @@ -240,16 +241,14 @@ public struct IOSSimRecordVideoCommand: SimUseExecutableCommand { let frameStart = Date() do { - let frameData = try await VideoFrameUtilities.captureScreenshotData(from: simulator) - if let cgImage = VideoFrameUtilities.makeCGImage(from: frameData) { - let now = Date() - var presentationTime = CMTime(seconds: now.timeIntervalSince(writerStartTime), preferredTimescale: 600) - if presentationTime <= lastPresentationTime { - presentationTime = CMTimeAdd(lastPresentationTime, CMTime(value: 1, timescale: 600)) - } - try recorder.append(image: cgImage, presentationTime: presentationTime) - lastPresentationTime = presentationTime + let cgImage = try frameSource.currentFrame() + let now = Date() + var presentationTime = CMTime(seconds: now.timeIntervalSince(writerStartTime), preferredTimescale: 600) + if presentationTime <= lastPresentationTime { + presentationTime = CMTimeAdd(lastPresentationTime, CMTime(value: 1, timescale: 600)) } + try recorder.append(image: cgImage, presentationTime: presentationTime) + lastPresentationTime = presentationTime } catch let error as VideoWriterStallError { throw error } catch { diff --git a/Sources/iOSSimBackend/Verbs/IOSSimStreamVideoCommand.swift b/Sources/iOSSimBackend/Verbs/IOSSimStreamVideoCommand.swift index 042c8cf2..e91047a8 100644 --- a/Sources/iOSSimBackend/Verbs/IOSSimStreamVideoCommand.swift +++ b/Sources/iOSSimBackend/Verbs/IOSSimStreamVideoCommand.swift @@ -17,6 +17,22 @@ public struct IOSSimStreamVideoCommand: SimUseExecutableCommand { case raw case ffmpeg case bgra + + /// The container this format's consumers can decode, or nil for + /// `bgra`, which carries raw pixels rather than encoded frames. + func frameContainer(quality: Int) -> FrameContainer? { + switch self { + // Motion JPEG is JPEG by definition and by consumer: ffmpeg's + // `mpjpeg` demuxer rejects any other container. + case .mjpeg: .jpeg(quality: quality) + // `raw` delimits frames with its own length prefix, and the + // documented `ffmpeg` pipeline sniffs the container + // (`-f image2pipe`), so both carry the capture losslessly and + // untranscoded. + case .raw, .ffmpeg: .png + case .bgra: nil + } + } } /// Summary of a completed stream run. The actual video bytes are @@ -50,7 +66,7 @@ public struct IOSSimStreamVideoCommand: SimUseExecutableCommand { @Option(help: "Frames per second (1-30, default: 10)") public var fps: Int = 10 - @Option(help: "JPEG quality (1-100, default: 80)") + @Option(help: "JPEG quality for mjpeg (1-100, default: 80). Ignored by raw/ffmpeg, which carry lossless PNG frames.") public var quality: Int = 80 @Option(help: "Scale factor (0.1-1.0, default: 1.0)") @@ -132,7 +148,15 @@ public struct IOSSimStreamVideoCommand: SimUseExecutableCommand { // frame counts; return a zero-summary so `format(_:)` emits nothing. return ExecutionResult(framesStreamed: 0, durationSeconds: 0, format: format) default: - return try await streamCompressedFrames(from: targetSimulator, format: format, cancellationFlag: cancellationFlag) + guard let container = format.frameContainer(quality: quality) else { + throw CLIError(errorDescription: "Format \(format.rawValue) does not carry encoded frames") + } + return try await streamCompressedFrames( + from: targetSimulator, + format: format, + container: container, + cancellationFlag: cancellationFlag + ) } } @@ -141,12 +165,14 @@ public struct IOSSimStreamVideoCommand: SimUseExecutableCommand { private func streamCompressedFrames( from simulator: FBSimulator, format: OutputFormat, + container: FrameContainer, cancellationFlag: CancellationFlag ) async throws -> ExecutionResult { FileHandle.standardError.write(Data("Starting screenshot-based video stream from simulator \(simulator.udid)...\n".utf8)) - FileHandle.standardError.write(Data("Format: \(format.rawValue), FPS: \(fps), Quality: \(quality), Scale: \(scale)\n".utf8)) + FileHandle.standardError.write(Data("Format: \(format.rawValue), FPS: \(fps), Frames: \(container), Scale: \(scale)\n".utf8)) FileHandle.standardError.write(Data("Press Ctrl+C to stop streaming\n".utf8)) + let frameSource = try await SimulatorFrameSource(simulator: simulator) let frameInterval = 1.0 / Double(fps) let mjpegBoundary = "--mjpegstream" let destination = FileHandle.standardOutput @@ -170,12 +196,15 @@ public struct IOSSimStreamVideoCommand: SimUseExecutableCommand { let frameStartTime = Date() do { - let screenshotData = try await VideoFrameUtilities.captureScreenshotData(from: simulator) - let processedData = try await VideoFrameUtilities.processJPEGData(screenshotData, scale: scale, quality: quality) + let processedData = try VideoFrameUtilities.encodeFrame( + frameSource.currentFrame(), + as: container, + scale: scale + ) switch format { case .mjpeg: - let frameHeader = "\(mjpegBoundary)\r\nContent-Type: image/jpeg\r\nContent-Length: \(processedData.count)\r\n\r\n" + let frameHeader = "\(mjpegBoundary)\r\nContent-Type: \(container.mimeType)\r\nContent-Length: \(processedData.count)\r\n\r\n" destination.write(Data(frameHeader.utf8)) destination.write(processedData) destination.write(Data("\r\n".utf8)) diff --git a/Tests/AndroidStreamVideoTests.swift b/Tests/AndroidStreamVideoTests.swift index aa4bd4c0..4f98e497 100644 --- a/Tests/AndroidStreamVideoTests.swift +++ b/Tests/AndroidStreamVideoTests.swift @@ -15,18 +15,22 @@ struct AndroidStreamVideoTests { #expect(result.stdout.starts(with: [0x00, 0x00, 0x00, 0x01])) } - @Test("mjpeg stream carries iOS-parity multipart framing") + @Test("mjpeg stream carries iOS-parity multipart framing, with JPEG frames") func mjpegSmoke() async throws { let result = try await streamForDuration(format: "mjpeg", duration: 4.0) #expect(isAcceptableStreamExitCode(result.exitCode), "Unexpected exit code: \(result.exitCode)") - let text = String(decoding: result.stdout.prefix(4096), as: UTF8.self) - #expect(text.contains("--mjpegstream")) - #expect(text.contains("Content-Type: image/jpeg")) #expect(result.stderr.contains("Format: mjpeg")) + + // `screencap` only emits PNG, so this is the one Android format that + // has to transcode — assert it actually did. + let frame = try firstMJPEGFrame(in: result.stdout) + #expect(frame.contentType == "image/jpeg") + #expect(frame.contentLength == frame.payload.count) + #expect(frame.payload.starts(with: [0xFF, 0xD8]), "frame payload is not JPEG") } - @Test("raw format prefixes each frame with a 4-byte length") + @Test("raw format prefixes each untranscoded PNG frame with a 4-byte length") func rawFraming() async throws { let result = try await streamForDuration(format: "raw", duration: 3.0) @@ -37,6 +41,8 @@ struct AndroidStreamVideoTests { // room to carry (screencap frames are tens of KB to a few MB). #expect(length > 100) #expect(Int(length) <= result.stdout.count) + // `screencap -p` output reaches stdout with no re-encode. + #expect(result.stdout.dropFirst(4).starts(with: [0x89, 0x50, 0x4E, 0x47]), "frame payload is not PNG") } @Test("h264 stream survives a screenrecord segment restart") diff --git a/Tests/MJPEGTestSupport.swift b/Tests/MJPEGTestSupport.swift new file mode 100644 index 00000000..fd471ff7 --- /dev/null +++ b/Tests/MJPEGTestSupport.swift @@ -0,0 +1,51 @@ +// SPDX-License-Identifier: Apache-2.0 +import Foundation +import Testing + +struct MJPEGTestFrame { + let contentType: String + let contentLength: Int + let payload: Data +} + +func firstMJPEGFrame(in stream: Data) throws -> MJPEGTestFrame { + let headerTerminator = Data("\r\n\r\n".utf8) + let outerHeader = try #require( + stream.range(of: headerTerminator), + "MJPEG stream is missing its outer HTTP header" + ) + let boundary = Data("--mjpegstream\r\n".utf8) + let partBoundary = try #require( + stream.range(of: boundary, in: outerHeader.upperBound.. String? { + let prefix = "\(name):" + return headerText.components(separatedBy: "\r\n") + .first { $0.hasPrefix(prefix) } + .map { String($0.dropFirst(prefix.count)).trimmingCharacters(in: .whitespaces) } + } + + let contentType = try #require(headerValue("Content-Type"), "MJPEG frame is missing Content-Type") + let lengthText = try #require(headerValue("Content-Length"), "MJPEG frame is missing Content-Length") + let contentLength = try #require(Int(lengthText), "MJPEG frame has an invalid Content-Length") + let payloadStart = partHeader.upperBound + let nextBoundaryPrefix = Data("\r\n--mjpegstream".utf8) + let nextBoundary = try #require( + stream.range(of: nextBoundaryPrefix, in: payloadStart.. 8) + let length = result.data.prefix(4).reduce(UInt32(0)) { ($0 << 8) | UInt32($1) } + #expect(Int(length) <= result.data.count) + #expect(result.data.dropFirst(4).starts(with: [0x89, 0x50, 0x4E, 0x47]), "frame payload is not PNG") } @Test("Stream video with custom FPS") @@ -49,7 +62,7 @@ struct StreamVideoTests { ) #expect(isAcceptableStreamExitCode(result.exitCode), "Unexpected exit code: \(result.exitCode)") - #expect(result.output.contains("Quality: 50")) + #expect(result.output.contains("Frames: jpeg q50")) #expect(result.output.contains("Scale: 0.5")) } diff --git a/Tests/VideoFrameProcessingTests.swift b/Tests/VideoFrameProcessingTests.swift index e11d06e4..7b43c000 100644 --- a/Tests/VideoFrameProcessingTests.swift +++ b/Tests/VideoFrameProcessingTests.swift @@ -5,9 +5,9 @@ import AppKit import CoreGraphics import SimUseVideo -/// Unit coverage for the frame-processing utilities behind the JPEG -/// streaming formats and the screencap-based recording fallbacks — -/// pure image plumbing, no device needed. +/// Unit coverage for the frame-processing utilities behind the streaming +/// formats and the screencap-based recording fallbacks — pure image +/// plumbing, no device needed. @Suite("VideoFrameUtilities — frame processing") struct VideoFrameProcessingTests { /// A solid-color PNG generated in-process, standing in for a @@ -55,6 +55,13 @@ struct VideoFrameProcessingTests { return try #require(rep.representation(using: .png, properties: [:])) } + private func makeImage(width: Int, height: Int) throws -> CGImage { + try #require(VideoFrameUtilities.makeCGImage(from: try makePNG(width: width, height: height))) + } + + private static let pngMagic = Data([0x89, 0x50, 0x4E, 0x47]) + private static let jpegMagic = Data([0xFF, 0xD8]) + @Test("makeCGImage decodes PNG data and rejects garbage") func makeCGImage() throws { let png = try makePNG(width: 64, height: 48) @@ -88,47 +95,106 @@ struct VideoFrameProcessingTests { #expect(tiny.height >= 2) } - @Test("processJPEGData passes data through untouched at default settings") - func processPassthrough() async throws { - // Pins the intentional fast path shared with iOS streaming: at - // scale 1.0 / quality 80 the frame is forwarded byte-for-byte - // (no decode/re-encode), whatever its container format. + // MARK: - encodeFrame: the single pass a captured frame pays + + @Test("encodeFrame writes the container it was asked for") + func encodeFrameContainers() throws { + let image = try makeImage(width: 64, height: 48) + + let png = try VideoFrameUtilities.encodeFrame(image, as: .png, scale: 1.0) + #expect(png.starts(with: Self.pngMagic)) + + let jpeg = try VideoFrameUtilities.encodeFrame(image, as: .jpeg(quality: 80), scale: 1.0) + #expect(jpeg.starts(with: Self.jpegMagic)) + + for data in [png, jpeg] { + let decoded = try #require(VideoFrameUtilities.makeCGImage(from: data)) + #expect(decoded.width == 64) + #expect(decoded.height == 48) + } + } + + @Test("encodeFrame honours JPEG quality") + func encodeFrameQuality() throws { + let image = try #require(VideoFrameUtilities.makeCGImage(from: try makePatternPNG(width: 128, height: 96))) + let low = try VideoFrameUtilities.encodeFrame(image, as: .jpeg(quality: 20), scale: 1.0) + let high = try VideoFrameUtilities.encodeFrame(image, as: .jpeg(quality: 90), scale: 1.0) + #expect(low.count < high.count) + } + + @Test("encodeFrame scales within the same pass, in either container") + func encodeFrameScales() throws { + let image = try makeImage(width: 100, height: 60) + + for container in [FrameContainer.png, .jpeg(quality: 80)] { + let out = try VideoFrameUtilities.encodeFrame(image, as: container, scale: 0.5) + let scaled = try #require(VideoFrameUtilities.makeCGImage(from: out)) + #expect(scaled.width == 50) + #expect(scaled.height == 30) + } + } + + // MARK: - transcodeFrame: for captures that cannot pick their container + + @Test("transcodeFrame passes a matching lossless container through byte-for-byte") + func transcodePassthrough() async throws { + // The zero-transcode path: `screencap -p` already emits exactly what + // the PNG-carrying formats want, so the bytes must not be touched. let png = try makePNG(width: 32, height: 32) - let out = try await VideoFrameUtilities.processJPEGData(png, scale: 1.0, quality: 80) + let out = try VideoFrameUtilities.transcodeFrame(png, to: .png, scale: 1.0) #expect(out == png) } - @Test("non-default quality re-encodes to JPEG") - func processReencodesQuality() async throws { - let png = try makePNG(width: 64, height: 64) - let out = try await VideoFrameUtilities.processJPEGData(png, scale: 1.0, quality: 50) - #expect(out != png) - #expect(out.prefix(2) == Data([0xFF, 0xD8])) - let reencoded = try #require(VideoFrameUtilities.makeCGImage(from: out)) - #expect(reencoded.width == 64) - #expect(reencoded.height == 64) + @Test("transcodeFrame encodes PNG input into JPEG when the sink needs JPEG") + func transcodeToJPEG() throws { + // Regression for MJPEG frames carrying PNG payloads under an + // `image/jpeg` header at the default settings. + let png = try makePNG(width: 32, height: 32) + let out = try VideoFrameUtilities.transcodeFrame(png, to: .jpeg(quality: 80), scale: 1.0) + #expect(out.starts(with: Self.jpegMagic)) + let decoded = try #require(VideoFrameUtilities.makeCGImage(from: out)) + #expect(decoded.width == 32) + #expect(decoded.height == 32) } - @Test("JPEG quality changes encoded output size") - func processAppliesQuality() async throws { - let png = try makePatternPNG(width: 128, height: 96) - let low = try await VideoFrameUtilities.processJPEGData(png, scale: 1.0, quality: 20) - let high = try await VideoFrameUtilities.processJPEGData(png, scale: 1.0, quality: 90) + @Test("transcodeFrame never passes JPEG through, so --quality is always honoured") + func transcodeAlwaysReencodesJPEG() throws { + // JPEG bytes carry no record of the quality they were encoded at, + // so a JPEG sink has to encode rather than trust the input. + let image = try #require(VideoFrameUtilities.makeCGImage(from: try makePatternPNG(width: 128, height: 96))) + let source = try VideoFrameUtilities.encodeFrame(image, as: .jpeg(quality: 95), scale: 1.0) - #expect(low != high) - #expect(low.count < high.count) + let out = try VideoFrameUtilities.transcodeFrame(source, to: .jpeg(quality: 20), scale: 1.0) + #expect(out != source) + #expect(out.count < source.count) } - @Test("scaling re-encodes to JPEG at the requested pixel dimensions") - func processScales() async throws { + @Test("transcodeFrame re-encodes a scaled frame back into its own container") + func transcodeScales() throws { let png = try makePNG(width: 100, height: 60) - let out = try await VideoFrameUtilities.processJPEGData(png, scale: 0.5, quality: 80) - #expect(out.prefix(2) == Data([0xFF, 0xD8])) + let out = try VideoFrameUtilities.transcodeFrame(png, to: .png, scale: 0.5) + #expect(out.starts(with: Self.pngMagic)) let scaled = try #require(VideoFrameUtilities.makeCGImage(from: out)) #expect(scaled.width == 50) #expect(scaled.height == 30) } + @Test("mimeType matches the bytes each container actually produces") + func mimeTypeMatchesPayload() throws { + // The bug this contract exists to prevent: a frame header that + // advertises a container the payload isn't. + let image = try makeImage(width: 32, height: 32) + + for container in [FrameContainer.png, .jpeg(quality: 80)] { + let data = try VideoFrameUtilities.encodeFrame(image, as: container, scale: 1.0) + let expectedMagic = container == .png ? Self.pngMagic : Self.jpegMagic + #expect(data.starts(with: expectedMagic), "\(container.mimeType) payload has the wrong magic bytes") + } + + #expect(FrameContainer.png.mimeType == "image/png") + #expect(FrameContainer.jpeg(quality: 80).mimeType == "image/jpeg") + } + @Test("estimateBitrate clamps to its floor and ceiling and grows with quality") func estimateBitrate() { // Tiny frame → floor.