Dataset snapshots cannot verify table or copied-media integrity after transfer

[Galabavamsi]

Current behavior

Dataset snapshot format v1 identifies the format, version, media mode, media URI base, and table filenames in format.json. It does not record file sizes, row counts, hashes, or a copied-asset inventory, and HFlow has no snapshot verification API or CLI command.

The atomic staging/activation path proves that HFlow finished writing a snapshot. It does not let a recipient distinguish those original bytes from a snapshot damaged or changed after export.

Controlled result

Tested at 81a13f81f5cf97d0b3048f0a660e3a1e701d50cb (v0.2.5-3-g81a13f8) with a one-episode, two-measurement, one-tag snapshot exported through the public API in copy media mode.

The original marker contained exactly:

format
format_version
media_mode
media_uri_base
tables

I made three independent copies and changed one thing in each:

Mutation after export	format.json changed?	Current observation
Replaced the copied media bytes	No	All Parquet tables still read normally; there is no signal that the asset changed
Deleted tags.parquet	No	The destination is still recognized as an overwriteable HFlow snapshot; DuckDB later raises “No files found”
Truncated samples.parquet to 64 bytes	No	The destination is still recognized as an overwriteable HFlow snapshot; DuckDB later raises “No magic bytes found at end of file”

hasattr(hflow, "verify_dataset_snapshot") was false, and hflow export snapshot --help exposes export options but no verification path.

This means the format marker can identify a snapshot while being unable to say whether its required tables and copied assets are the bytes HFlow actually published.

Why this matters

Snapshots are the delivery artifact intended to move independently of the catalog. A successful export followed by an interrupted copy, storage corruption, or accidental asset replacement currently looks valid until a consumer happens to read the affected file. Modified copied media may remain completely silent.

This is also consistent with adjacent dataset-format practice: Croissant strongly recommends SHA-256 checksums for every FileObject in a versioned dataset, and Hugging Face Datasets verification covers expected files, split metadata, and optionally checksums.

• https://docs.mlcommons.org/croissant/docs/croissant-spec.html
• https://huggingface.co/docs/datasets/about_dataset_load

Design questions

A v2 marker plus a public verifier appears to be one possible shape, but the format contract should be decided before implementation:

1. Should every required table record path, size_bytes, sha256, and perhaps row_count?
2. In copy mode, should every copied asset receive the same coverage, or should the marker point to a separate deterministic asset manifest?
3. Should there also be one digest over the normalized manifest so that the delivered set itself has an identity?
4. How should v1 behave: recognizable but explicitly “unverifiable”, or rejected by a strict verifier?
5. In references mode, should verification cover only local tables/metadata and report remote media as unverified unless network access is explicitly requested?
6. Are unlisted extra files allowed?

Definition of done

• Every required Parquet table in a newly exported snapshot is covered by deterministic integrity metadata.
• Every local asset in copy mode is covered.
• A public library and/or CLI verification path detects a missing file, truncation, and same-path byte replacement.
• An unchanged snapshot passes.
• v1 compatibility and reference-mode semantics are deliberate and tested.
• Default verification does not fetch remote media.
• The snapshot format documentation explains what is and is not guaranteed.

Non-goals

• Cryptographic signing or publisher authenticity.
• Proving continued availability of remote reference-mode media.
• Changing the logical contents of snapshot tables.
• Making the whole catalog content-addressed.

[kstonekuan]

Accepted, labelled enhancement / advanced. The three-mutation table is the right way to show it, particularly the media row: replacing copied asset bytes leaves nothing detectable anywhere, while the two Parquet mutations at least surface eventually as a DuckDB error a consumer can act on.

Direction, and it splits the way #379 did. Recording integrity in the marker is cheap and almost free of downside. A verifier is a product commitment: it makes HFlow answerable for delivery integrity, including what it means when verify fails on a corpus someone paid for. So do the marker, leave the verifier for a separate decision, and make sure the marker carries everything a verifier will need so that it can arrive purely additively rather than as a second format change.

Do not bump format_version. I checked the reader and it is not strict about the key set:

snapshot.py:521-522   format_marker.get("format") != DATASET_SNAPSHOT_FORMAT_NAME
                      or format_marker.get("format_version") != DATASET_SNAPSHOT_FORMAT_VERSION

Only those two keys are compared, and nothing rejects unexpected fields. So added integrity keys are backward compatible as they stand. Bumping to v2 would be the opposite of harmless: that comparison is in the overwrite path, so a v1 HFlow would refuse to replace a v2 snapshot directory. Adding keys under v1 is strictly better than a version break, and it is available to you for free.

On your three design questions:

1. path, size_bytes, and sha256 per required table, yes. row_count no, or not as an integrity field: it is derived from content rather than independent of it, so it duplicates what the hash already covers while adding a value that can disagree with the file for reasons that are not corruption.
2. Same coverage inline, not a separate asset manifest. A second file is a second thing that can go missing, and the failure you found silent is exactly the copied media, so it should not be the part behind an extra indirection.
3. Yes to one digest over the normalized set, and this is the item worth most. Per-file hashes catch a changed file; only a digest over the whole normalized inventory catches a deleted one, which is your second mutation row. Note the importer settled the same question for prepared-manifest.json in feat(import): per-episode receipts in prepared-manifest.json #389 and left verify deferred, so keeping the two artifacts' receipt shapes recognizably similar is worth a little effort.

Croissant and the HF datasets verification docs are the right prior art to cite. Worth being explicit in the PR about one difference: those describe verification of a published dataset by a consumer, and what lands here is only the record that makes such verification possible later.

Non-goal for this issue: no CLI command, no verify_dataset_snapshot, no verification during export. If the marker is right, all three of those are readers of it.