test: Add failing Harness test — square Frame region maps to a 3.16:1 strip in View space on iOS

[florianpreusner]

Adds one failing Harness test reproducing an iOS-only anisotropy when composing the two public coordinate conversions. Per the harness README, this PR is the reproduction — the issue will reference it.

What's wrong

Composing the library's own two transforms produces a distorted mapping on iOS:

PreviewView.convertViewPointToCameraPoint()  ->  Frame.convertCameraPointToFramePoint()

A square region — the natural thing to map when you have a scan guide or a detection box — comes back as a wide, short strip. Each sample smears across its neighbours.

Measured

iPhone 15 Pro Max, iOS 26.5.2, frame 3840×2160, orientation: "left". A square guide cell, identical ±19.7 dp in both view axes:

sensor Δx 0.0562  ->  frame Δy 121.4 px
sensor Δy 0.0999  ->  frame Δx 383.6 px

383.6 × 121.4 px — ratio 3.159, where a correct mapping gives 1.000. Pixel 8 (1280×720) composes correctly at 1.000, so this is iOS-only.

convertViewPointToCameraPoint looks right on its own: for the square patch it returns Δx/Δy = 0.5625 = exactly 9/16, consistent with per-axis normalisation over a 16:9 image.

Suspected cause

ios/Utils/FrameCoordinateSystemConverter.getFrameToCameraMatrix rotates within [0, 1]² in step 1, then normalises in step 3:

// 1. Counter-rotate by the orientation to get it up-right
case .left: matrix = matrix.translatedBy(x: 1, y: 0).rotated(by: .pi / 2)
...
// 3. Our Matrix is in [0, 1], so let's scale it to [0, width|height] now
matrix = matrix.scaledBy(x: 1 / width, y: 1 / height)

scaledBy post-concatenates, so a point is normalised by the buffer's own width/height before the rotation swaps the axes — and the extents never swap with it. The measured numbers match that reading exactly: 0.0562 × 2160 = 121.4 and 0.0999 × 3840 = 383.6, i.e. each output axis is scaled by its own extent after the swap, where the swap should have transposed them. Predicted anisotropy for a 16:9 frame at orientation: left is (16/9)² = 3.1605, against the measured 3.159.

One candidate shape for the fix, though the test is the arbiter:

switch orientation {
case .left, .right: matrix = matrix.scaledBy(x: 1 / height, y: 1 / width)
case .up, .down:    matrix = matrix.scaledBy(x: 1 / width, y: 1 / height)
}

Corner and round-trip behaviour stay correct under the bug, which is why the existing tests don't catch it:

• round-trips Frame -> Camera -> Frame passes because the error is self-inverse — getCameraToFrameMatrix is literally getFrameToCameraMatrix().inverted().
• round-trips Frame center -> Camera -> View center end-to-end passes because the frame center maps to camera (0.5, 0.5) under both a correct and an anisotropic matrix, so it lands on the view center either way. The center is a fixed point of the error.

Only an off-center region exposes it.

The test

maps a square Frame region onto a square View region in visioncamera.coordinates.harness.tsx, next to the existing end-to-end test. It runs the square the other way round — Frame → Camera → View — so the frame callback is installed before start() exactly like the neighbouring tests, and no new setup shape is introduced.

• Takes a square centered in the frame, side min(width, height) / 4, kept small so it stays inside the preview's resizeMode='cover' crop on any view aspect ratio.
• Converts three corners to camera space in the worklet, then to view space through the mounted PreviewView.
• Compares the two adjacent edge lengths, not per-axis deltas — the orientation counter-rotation is free to map the frame's x edge onto the view's y axis, so only distances are comparable.
• Asserts leftEdge / topEdge is toBeCloseTo(1, 1) — a ratio rather than absolute lengths, since frame resolution and view size differ per device, but a square is square everywhere.

No platform guard, per README §8: the behaviour should hold on both platforms, so the shared test should show the discrepancy rather than hide it. Expect red on iOS, green on Android.

What I could not verify myself

I have not executed this test. I don't have a rig for this monorepo (no bun, and the failure is iOS-only), so the CI run on this PR is the first real execution. The test is written against the conventions in __tests__/README.md and biome check is clean, but treat the exact tolerance as a starting point — if toBeCloseTo(1, 1) proves too tight for real device geometry on a correct implementation, it should be loosened rather than the invariant weakened.

Also worth flagging: fork PRs may not be able to assume the AWS Device Farm OIDC role, so the iOS leg might not run here without a maintainer-triggered run.

Context

Found while mapping a Rubik's cube scan guide onto sampled frames in a shipping app. Our workaround validates the composed mapping with an isotropy check and falls back to deriving the geometry from the frame dimensions when it fails — no platform branch, it's measured at runtime. Android accepts the library transform; iOS rejects it at 3.159 and falls back. Happy to adjust anything here, and glad to re-run against a fix.

[vercel]

Someone is attempting to deploy a commit to the Margelo Team on Vercel.

A member of the Team first needs to authorize it.

[mrousavy]

Danke!