LeRobot depth videos are accepted as RGB and lose 12-bit depth semantics

[Galabavamsi]

Current behavior

The LeRobot importer discovers every feature with dtype: video as a selectable camera. It does not inspect the feature info block for is_depth_map (or the legacy video.is_depth_map spelling).

Every selected video then takes the RGB transform path:

source MP4 -> libx264 -crf 23 -> foxglove.CompressedVideo

That is unsafe for current LeRobot depth streams. LeRobot stores depth as quantized 12-bit codes in a high-bit-depth video and persists the parameters needed to dequantize those codes to physical units. HFlow neither preserves those fields nor rejects the unsupported semantic type.

The user-facing guide currently says depth maps are unsupported, and the public API says unsupported feature layouts fail before any episode is published. A dtype: video depth feature bypasses that refusal.

Controlled result

Tested at 81a13f81f5cf97d0b3048f0a660e3a1e701d50cb.

I generated four 32 by 24 constant depth-code frames with 12-bit values 64, 1024, 2048, and 4032, encoded them losslessly as the shape LeRobot documents for video depth, then passed the MP4 through the importer transform _transcode_mp4_to_h264.

source:    codec=hevc  pix_fmt=gray12le      frames=4
converted: codec=h264  pix_fmt=yuv420p10le

source decoded medians:    [1024, 16388, 32776, 64527]
converted decoded medians: [1025, 16400, 32767, 64510]
source unique values/frame:    [1, 1, 1, 1]
converted unique values/frame: [1, 1, 2, 1]
exact_equal: false

The decoded values are represented as 16-bit output values in this comparison. The relevant outcome is that a lossless 12-bit source does not survive the current CRF-23 H.264 route exactly, and the output stream has only a 10-bit pixel format.

Independently, the current metadata discovery is:

feature_specification.get("dtype") == "video"

with no depth-semantic guard before conversion.

Real-format evidence

Current LeRobot documentation specifies that depth maps may be stored as video streams alongside RGB, quantized to 12-bit codes and encoded as HEVC Main 12. It records is_depth_map: true plus video.depth_min, video.depth_max, video.shift, and video.use_log so readers can recover physical units:

• https://github.com/huggingface/lerobot/blob/main/docs/source/video_encoding_parameters.mdx
• https://github.com/huggingface/lerobot/blob/main/src/lerobot/datasets/dataset_metadata.py

There is also a pinned public official v3 depth corpus showing the active metadata contract for image-backed depth:

• lerobot/outdoor-depth
• revision 7c63dc16226035cdd79fffc95404e82723bda5ad
• feature observation.images.depth
• dtype: image, is_depth_map: true, depth_unit: mm

That corpus is evidence for the semantic marker, not a video acceptance fixture.

Immediate safe fix

Until HFlow has an agreed canonical depth representation, selecting a video feature marked as depth should fail before any episode or manifest is published. The error should name the feature and explain that the RGB H.264 path cannot preserve depth values.

Both current and legacy LeRobot marker spellings should be considered:

• info.is_depth_map
• info["video.is_depth_map"]
• video_info["video.is_depth_map"]

Advanced extension to agree separately

End-to-end support needs a format decision rather than merely allowing the feature through:

1. Decode LeRobot quantized depth with its persisted parameters into a named physical unit, or deliberately preserve the codes plus their calibration metadata.
2. Choose a lossless canonical channel representation suitable for one-channel 16-bit integer and/or 32-bit floating-point depth.
3. Keep depth out of RGB luma, exposure, and blackout checks unless a depth-specific adapter is defined.
4. Preserve depth provenance through catalog and snapshot export.
5. Validate on a real public video-depth v3 corpus once one with state/action and stable pinned metadata is selected.

Definition of done for this issue

• A selected depth-marked video cannot silently take the RGB H.264 transform path.
• Both modern and legacy marker spellings are covered.
• Refusal happens before a partial episode or prepared manifest is published.
• A normal RGB video remains accepted.
• Documentation names the refusal and points to the metadata field.
• If maintainers choose full depth support instead, exact depth values and physical-unit metadata round-trip in an outcome-focused test.

Prior scope

#189 explicitly left depth maps out of scope unless a small extension was proposed and agreed. This issue is that extension plus the currently missing fail-loud boundary; it does not reopen the generalized RGB importer work.