Symptom
stream-video --scale 0.5 (JPEG formats: mjpeg / raw / ffmpeg) is supposed to shrink each frame to half its dimensions to save bandwidth and encode time. On a Retina Mac it effectively does nothing to the pixel count: the emitted JPEG has (roughly) the same pixel dimensions as the unscaled frame — you pay the full decode + re-encode cost per frame and get none of the size reduction. On a non-Retina display (or a headless CI host) the same command produces genuinely half-size frames. Same flags, different output, depending on the Mac it runs on.
This was noticed while unit-testing the frame pipeline for #81: scaling a 100×60 test frame at --scale 0.5 produced a 100×60 JPEG on a Retina host instead of the expected 50×30. The current test deliberately pins only "re-encoded + aspect ratio preserved" because the absolute pixel size is environment-dependent.
Root cause
VideoFrameUtilities.scaleJPEGData (Sources/SimUseVideo/VideoCommandSupport.swift) resizes through AppKit:
let newImage = NSImage(size: newSize) // newSize is in POINTS
newImage.lockFocus() // offscreen context at the MAIN SCREEN's backing scale
image.draw(in: NSRect(origin: .zero, size: newSize))
newImage.unlockFocus()
Two unit systems are being conflated:
NSImage.size is in points, not pixels.
lockFocus() renders into an offscreen context whose backing scale factor follows the host's main screen — 2x on Retina. Drawing a "50×30 point" image therefore produces a 100×60 pixel bitmap, which tiffRepresentation then encodes.
So the point-space math is correct, but the pixel output is points × hostBackingScale — on a 2x host, --scale 0.5 round-trips back to the original pixel count. reencodeJPEGData (the quality-only path) has the same AppKit dependency but no size math, so it is only exposed to DPI-metadata edge cases, not the scale bug.
Affected / not affected
| Surface |
Affected? |
stream-video --scale JPEG formats, iOS (IOSSimStreamVideoCommand.swift:174) |
Yes |
stream-video --scale JPEG formats, Android (AndroidStreamVideoCommand.swift:321) |
Yes (shares the same helper) |
stream-video --format h264 (Android) |
No — scale maps to screenrecord --size in pixels |
record-video, both platforms |
No — native FBSimulatorVideoStream scaleFactor / screenrecord --size; the screencap fallback draws into a pixel-exact CGContext via H264StreamRecorder |
screenshot |
No — no scaling applied |
Proposed fix
Replace the AppKit resize with a pixel-deterministic CoreGraphics/ImageIO pipeline, reusing what the file already has:
makeCGImage(from:) → source CGImage (pixel dimensions, no DPI ambiguity).
computeDimensions(for:scale:) → target pixel size (already exists, already unit-tested, already enforces even dimensions).
- Draw into a
CGContext of exactly those pixel dimensions (same pattern as H264StreamRecorder.append).
- Encode with
CGImageDestination + kCGImageDestinationLossyCompressionQuality (also replaces the NSImage/TIFF round-trip in reencodeJPEGData, which is an extra full-image copy per frame today).
Benefits: --scale means the same thing on every host, the AppKit dependency (and its main-screen coupling) drops out of SimUseVideo, and the per-frame TIFF intermediate goes away.
Test plan
- Tighten
VideoFrameProcessingTests.processScales from "re-encoded + aspect preserved" to exact expected pixel dimensions (100×60 @ 0.5 → 50×30) — it becomes the regression pin once output is deterministic.
- Add a quality-path assertion that re-encode preserves exact input dimensions.
- Live sanity:
stream-video --format ffmpeg --scale 0.5 on a Retina host, decode a frame, confirm halved pixel dimensions and a visibly smaller per-frame byte size.
Refs: #78, #81 (where the behavior was pinned as-is).
Symptom
stream-video --scale 0.5(JPEG formats:mjpeg/raw/ffmpeg) is supposed to shrink each frame to half its dimensions to save bandwidth and encode time. On a Retina Mac it effectively does nothing to the pixel count: the emitted JPEG has (roughly) the same pixel dimensions as the unscaled frame — you pay the full decode + re-encode cost per frame and get none of the size reduction. On a non-Retina display (or a headless CI host) the same command produces genuinely half-size frames. Same flags, different output, depending on the Mac it runs on.This was noticed while unit-testing the frame pipeline for #81: scaling a 100×60 test frame at
--scale 0.5produced a 100×60 JPEG on a Retina host instead of the expected 50×30. The current test deliberately pins only "re-encoded + aspect ratio preserved" because the absolute pixel size is environment-dependent.Root cause
VideoFrameUtilities.scaleJPEGData(Sources/SimUseVideo/VideoCommandSupport.swift) resizes through AppKit:Two unit systems are being conflated:
NSImage.sizeis in points, not pixels.lockFocus()renders into an offscreen context whose backing scale factor follows the host's main screen — 2x on Retina. Drawing a "50×30 point" image therefore produces a 100×60 pixel bitmap, whichtiffRepresentationthen encodes.So the point-space math is correct, but the pixel output is
points × hostBackingScale— on a 2x host,--scale 0.5round-trips back to the original pixel count.reencodeJPEGData(the quality-only path) has the same AppKit dependency but no size math, so it is only exposed to DPI-metadata edge cases, not the scale bug.Affected / not affected
stream-video --scaleJPEG formats, iOS (IOSSimStreamVideoCommand.swift:174)stream-video --scaleJPEG formats, Android (AndroidStreamVideoCommand.swift:321)stream-video --format h264(Android)screenrecord --sizein pixelsrecord-video, both platformsFBSimulatorVideoStream scaleFactor/screenrecord --size; the screencap fallback draws into a pixel-exactCGContextviaH264StreamRecorderscreenshotProposed fix
Replace the AppKit resize with a pixel-deterministic CoreGraphics/ImageIO pipeline, reusing what the file already has:
makeCGImage(from:)→ sourceCGImage(pixel dimensions, no DPI ambiguity).computeDimensions(for:scale:)→ target pixel size (already exists, already unit-tested, already enforces even dimensions).CGContextof exactly those pixel dimensions (same pattern asH264StreamRecorder.append).CGImageDestination+kCGImageDestinationLossyCompressionQuality(also replaces the NSImage/TIFF round-trip inreencodeJPEGData, which is an extra full-image copy per frame today).Benefits:
--scalemeans the same thing on every host, the AppKit dependency (and its main-screen coupling) drops out ofSimUseVideo, and the per-frame TIFF intermediate goes away.Test plan
VideoFrameProcessingTests.processScalesfrom "re-encoded + aspect preserved" to exact expected pixel dimensions (100×60 @ 0.5 → 50×30) — it becomes the regression pin once output is deterministic.stream-video --format ffmpeg --scale 0.5on a Retina host, decode a frame, confirm halved pixel dimensions and a visibly smaller per-frame byte size.Refs: #78, #81 (where the behavior was pinned as-is).