FORMAT.md documents five episode/v1 keys; the LeRobot importer writes eleven, and five of the undocumented ones gate import resume

[kstonekuan]

docs/FORMAT.md:104-115 documents episode/v1 as five named keys plus "(any user key)". The LeRobot importer writes eleven, and six of them are not user keys in any meaningful sense: they are a contract between the importer and the resume path that reads them back.

What the importer writes today, at src/hflow/importers/lerobot.py:1229-1249:

            episode_record: dict[str, str] = {
                "task": ...,                     # documented
                "operator": ...,                 # documented
                "embodiment": ...,               # documented
                "source_dataset": ...,           # NOT documented
                "source_revision": ...,          # NOT documented
                "source_episode_index": ...,     # NOT documented
                "converter_version": ...,        # NOT documented
                "camera_keys": ...,              # NOT documented
                "gop_seconds": ...,              # NOT documented
            }
            ...
                episode_record["success"] = ...            # documented
                episode_record["success_derivation"] = ... # NOT documented

$ grep -c source_episode_index docs/FORMAT.md   ->  0
$ grep -c camera_keys docs/FORMAT.md            ->  0
$ grep -c success_derivation docs/FORMAT.md     ->  0

Why this is worth writing down rather than shrugging at

These are not incidental. _episode_identity_matches at src/hflow/importers/lerobot.py:181-193 reads five of them back to decide whether an already-published landing file may be reused instead of reconverted (#303). A key nobody documented is now load-bearing for a correctness decision, and the person most likely to break it is someone who reads FORMAT.md, sees "any user key passes through untouched", and concludes these are free-form.

success_derivation is the one a data consumer actually needs, because it names the methodology behind success: max(stats/next.success) means any success frame makes the episode a success, which is a claim a buyer filtering on success should be able to see.

One subtlety to get right

gop_seconds appears in both records. FORMAT.md already documents it under provenance/v1 as "the keyframe interval actually used", and the importer now also writes it into episode/v1 so the resume check can compare it without opening the transform's record. Same name, same value, two records, two reasons. Say so explicitly; a reader who finds it in one place will assume the other is a mistake.

What to do

Add the six undocumented keys to the episode/v1 table in docs/FORMAT.md, and note which are written by the LeRobot importer rather than by a recorder. The existing sentence "All keys are optional; the record is copied/merged from the source recording" stays true and is worth keeping, but it currently reads as though nothing but a recorder ever writes this record.

Pattern to copy

The provenance/v1 table immediately below it at docs/FORMAT.md:117-126 is the right level of detail: one row per key, one clause of meaning, format stated where it is not obvious (camera_keys is a compact JSON array of strings, gop_seconds is %g-formatted).

Definition of done

1. Every key src/hflow/importers/lerobot.py:1229-1249 can write appears in the episode/v1 table.
2. Each row says enough to reproduce the value's format, not just its meaning.
3. gop_seconds appearing in two records is explained rather than duplicated silently.
4. success_derivation's relationship to success is stated: it names the derivation, and both are absent together when the source carries no outcome feature.
5. No source changes. If writing this up suggests a key is wrong or misnamed, say so on the issue rather than changing it here.

Validation

Markdown only, so the test suite is not affected. Run the link check from CONTRIBUTING.md:

lychee --no-progress --include-fragments \
  --exclude '^https://github\.com/Hebbian-Robotics/hflow/(issues|security/advisories/new)$' \
  --exclude-path references/mcap-spec.md \
  --exclude-path references/foxglove-CompressedVideo.proto .