Skip to content

fix: stream-video --scale renders via NSImage.lockFocus at the host's backing scale factor (no-op on Retina) #83

Description

@onevcat

Symptom

stream-video --scale 0.5 (JPEG formats: mjpeg / raw / ffmpeg) is supposed to shrink each frame to half its dimensions to save bandwidth and encode time. On a Retina Mac it effectively does nothing to the pixel count: the emitted JPEG has (roughly) the same pixel dimensions as the unscaled frame — you pay the full decode + re-encode cost per frame and get none of the size reduction. On a non-Retina display (or a headless CI host) the same command produces genuinely half-size frames. Same flags, different output, depending on the Mac it runs on.

This was noticed while unit-testing the frame pipeline for #81: scaling a 100×60 test frame at --scale 0.5 produced a 100×60 JPEG on a Retina host instead of the expected 50×30. The current test deliberately pins only "re-encoded + aspect ratio preserved" because the absolute pixel size is environment-dependent.

Root cause

VideoFrameUtilities.scaleJPEGData (Sources/SimUseVideo/VideoCommandSupport.swift) resizes through AppKit:

let newImage = NSImage(size: newSize)   // newSize is in POINTS
newImage.lockFocus()                    // offscreen context at the MAIN SCREEN's backing scale
image.draw(in: NSRect(origin: .zero, size: newSize))
newImage.unlockFocus()

Two unit systems are being conflated:

  1. NSImage.size is in points, not pixels.
  2. lockFocus() renders into an offscreen context whose backing scale factor follows the host's main screen — 2x on Retina. Drawing a "50×30 point" image therefore produces a 100×60 pixel bitmap, which tiffRepresentation then encodes.

So the point-space math is correct, but the pixel output is points × hostBackingScale — on a 2x host, --scale 0.5 round-trips back to the original pixel count. reencodeJPEGData (the quality-only path) has the same AppKit dependency but no size math, so it is only exposed to DPI-metadata edge cases, not the scale bug.

Affected / not affected

Surface Affected?
stream-video --scale JPEG formats, iOS (IOSSimStreamVideoCommand.swift:174) Yes
stream-video --scale JPEG formats, Android (AndroidStreamVideoCommand.swift:321) Yes (shares the same helper)
stream-video --format h264 (Android) No — scale maps to screenrecord --size in pixels
record-video, both platforms No — native FBSimulatorVideoStream scaleFactor / screenrecord --size; the screencap fallback draws into a pixel-exact CGContext via H264StreamRecorder
screenshot No — no scaling applied

Proposed fix

Replace the AppKit resize with a pixel-deterministic CoreGraphics/ImageIO pipeline, reusing what the file already has:

  1. makeCGImage(from:) → source CGImage (pixel dimensions, no DPI ambiguity).
  2. computeDimensions(for:scale:) → target pixel size (already exists, already unit-tested, already enforces even dimensions).
  3. Draw into a CGContext of exactly those pixel dimensions (same pattern as H264StreamRecorder.append).
  4. Encode with CGImageDestination + kCGImageDestinationLossyCompressionQuality (also replaces the NSImage/TIFF round-trip in reencodeJPEGData, which is an extra full-image copy per frame today).

Benefits: --scale means the same thing on every host, the AppKit dependency (and its main-screen coupling) drops out of SimUseVideo, and the per-frame TIFF intermediate goes away.

Test plan

  • Tighten VideoFrameProcessingTests.processScales from "re-encoded + aspect preserved" to exact expected pixel dimensions (100×60 @ 0.5 → 50×30) — it becomes the regression pin once output is deterministic.
  • Add a quality-path assertion that re-encode preserves exact input dimensions.
  • Live sanity: stream-video --format ffmpeg --scale 0.5 on a Retina host, decode a frame, confirm halved pixel dimensions and a visibly smaller per-frame byte size.

Refs: #78, #81 (where the behavior was pinned as-is).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions