feat: Add convertImageToFrame(...) to convert Images back to Frames - #4087
Closed
mrousavy wants to merge 18 commits into
Closed
feat: Add convertImageToFrame(...) to convert Images back to Frames#4087mrousavy wants to merge 18 commits into
convertImageToFrame(...) to convert Images back to Frames#4087mrousavy wants to merge 18 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
18 tasks
Mirrored Frames in landscape orientations (`right`/`left`) were rendered 180° rotated, because the shaders applied the mirror to the upright output coordinate before undoing the rotation. `isMirrored` mirrors the already-rotated buffer (matching UIImage orientation flag semantics like `.rightMirrored`), so the inverse mapping has to undo the mirror after the inverse rotation. Unmirrored and 0°/180° Frames are unaffected since those transforms commute. Caught by the new byte-level Resizer harness tests.
…ertImageToFrame(...)`
…ver quirks CPU-written flexible-YUV AHardwareBuffers are sampled with swapped chroma planes by some GPU drivers (Cb/Cr plane-order ambiguity of planar layouts), and emulators cannot sample them at all. RGBA has no plane-order ambiguity, is written byte-exactly via a software Canvas (no ImageWriter, no odd-dimension restrictions), and GPU pipelines support any AHardwareBuffer format on Android anyways. The Barcode Scanner Frame path now falls back to Bitmap-based MLKit input for non-YUV Frames (which also fixes scanning real `rgb` camera Frames), and `getPixelBuffer()` eagerly probes CPU-locking of HardwareBuffers so it can fall back to the plane-based paths on devices where locking fails (e.g. emulators).
react-native-nitro-image's Android `loadFromRawPixelData` packs ColorInt-ordered ints but raw-copies them into the RGBA-laid-out ARGB_8888 Bitmap memory, R/B-swapping every 4-byte format - which made the Resizer color tests read swapped channels. Its `BGRA` fast path raw-copies bytes 1:1, so the tests label their R,G,B,A buffers as `BGRA` on Android until nitro-image is fixed.
…Frame(...)` `convertImageToFrame(...)` now produces camera-like RGB Frames on both platforms (`rgb-bgra-8-bit` on iOS, `rgb-rgba-8-bit` on Android) instead of YUV 4:2:0 on iOS - the Image is RGB, so the Frame stays RGB and the conversion is lossless (no more BT.601 rounding, no even-dimension restrictions). The Metal Resizer now also accepts `rgb-bgra-8-bit` input Frames next to `yuv-420-8-bit-full`: the shader samples a single BGRA texture (selected via a per-dispatch uniform) instead of the Y+CbCr planes, so real `rgb` camera Frames can be resized on iOS too. The iOS Barcode Scanner falls back to UIImage-based MLKit input for non-YUV Frames, since MLKit mis-reads BGRA sample buffers with padded row strides (sheared image) - mirroring the Android Bitmap fallback.
…areBuffer size bug nitro-modules' `ArrayBuffer.wrap(HardwareBuffer)` computes its size as `height * stride`, but `AHardwareBuffer_Desc.stride` is measured in pixels - the bytes-per-pixel factor is missing, so wrapped RGBA buffers report ~1/4 of their actual size and stride-based indexing runs out of bounds. Read the (correctly-sized) Image plane buffer instead on Android; iOS BGRA Frames are non-planar and keep using getPixelBuffer().
…eness Instead of generic `primaryTexture`/`secondaryTexture` slots with a per-dispatch `inputFormat` uniform branch, each input format now has its own set of specialized kernel entry points (`resize_yuv_*` sampling `yTexture` + `uvTexture`, `resize_bgra_*` sampling `bgraTexture`) that share the resize/rotate/mirror logic via `outputToInputCoordinate()` - mirroring the structure of the Android GLSL shader. The right pipeline is picked per frame at encode time; both pipeline states are built once at `createResizer` time, so there is zero per-dispatch overhead (and no more placeholder texture binding).
The scale-mode mapping used the raw buffer dimensions, but rotating normalized coordinates of a non-square texture is not shape-preserving - 90°/270°-rotated Frames were center-cropped with the wrong aspect (anisotropic distortion in cover) and wrongly letterboxed (contain). Both shaders now flip the source size for sideways rotations so the fit happens in upright-content space; the existing inverse-rotation step maps the coordinate back to buffer space. The scale-mode harness tests now also run against 'right'-rotated non-square Frames, whose output must be identical to the upright case.
Load the precompiled metallib once per Resizer and reuse it for both input-format pipeline states, precompute each pipeline's threadgroup size at init instead of per dispatch, and dispatch uniform threadgroups (rounded up, the kernels bounds-check `gid` anyway) so the resizer also works on GPUs without non-uniform threadgroup support (< A11).
mrousavy
force-pushed
the
claude/frameconverter-image-conversion-ae54f3
branch
from
July 28, 2026 09:52
befbb1b to
991bb37
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds FrameConverter.convertImageToFrame(...) and its async counterpart on iOS and Android. It converts a Nitro Image into a lossless RGB Frame with the requested orientation and mirroring.
The main goal is deterministic, camera-less testing of Frame-based APIs. Tests can load image fixtures, convert them to Frames, and verify real output without pointing a physical camera at a test target. The API can also be used with still images outside tests.
Included