Skip to content

Add a round-trip integrity / lossless-verification arm (sample source vs converted) #37

Description

@emmanuelmathot

Motivation

The harness measures size, write, read and display, but asserts nothing about whether a conversion is lossless. For every arm (netCDF → COPC, netCDF → GeoZarr, GeoTIFF → COG, shapefile → GeoParquet) we should be able to prove the converted object actually carries the source's data. A CNES reviewer asked us to verify exactly this for the PIXC → COPC path ("verify we really have everything") at the 2026-06-24 check-in.

Proposed change

An optional integrity metric that, after the target is written, samples values from the source and the target at corresponding locations and asserts equality per variable within a tolerance. Report, per variable: checked yes/no, points/cells/features compared, max abs diff, max rel diff, pass/fail.

  • Sampling configurable: samples.integrity = N (random points/cells/features) or all.
  • Per-format comparator behind a small interface:
    • point cloud (COPC): match by point identity (index after the finite-mask, or nearest-coord), compare x/y/z + every extra dimension.
    • raster (COG / GeoZarr): compare a set of pixel windows against the source bands/subdatasets.
    • vector (GeoParquet): compare feature geometry + attributes by feature id.
  • Surfaces in result.json next to the other metrics; a human line in summary.md.

Acceptance

  • integrity appears in result.json with per-variable diffs.
  • CI exercises it on the synthetic fixture (lossless ⇒ pass), and a deliberately lossy path (e.g. dropping a variable) ⇒ fail.
  • Documented in docs/configuration.md (metrics) and docs/architecture.md (comparator seam).

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions