Embed provenance of pipeline inputs into output data as a CycloneDX-style data Bill of Materials

[brawer]

Summary

Right now, provenance of the pipeline's inputs only shows up in pipeline.log, not in the data we actually publish. import_osm logs the OSM planet dump's header fields (replication_timestamp, source, writingprogram, see #634), and the AllThePlaces run metadata (run_id, output_url, start_time/end_time, spiders, total_lines, ...) will similarly be logged once #639 lands -- but a downstream consumer who only has our output files has no way to answer "what inputs, and which exact snapshot of them, produced this file?" without going back to logs we don't even publish alongside the data.

The same gap exists for our own pipeline version: #588 tracks that CARGO_PKG_VERSION is only ever logged, never stamped into the output itself. That's really the same problem (provenance not surviving into the output), just for the "which code produced this" axis rather than the "which data went in" axis -- so it's folded into this issue as one of the steps below, rather than being a separate effort. gh issue close 588 as a duplicate once this lands, or keep it open just for the version-stamping step; your call when you get here.

What to do

Model all of this as a single JSON document per output file, shaped like a CycloneDX 1.7 Bill of Materials -- but describing data lineage rather than code dependencies (component type: "data", one component per input source, a properties array carrying the source-specific attributes). This repo already has an established CycloneDX-for-data convention for the build-time code SBOM (scripts/sbom/pipeline.jq, components id-tagging-schema / osm-testdata-grid); reuse that same shape here, for consistency:

{
  "bomFormat": "CycloneDX",
  "specVersion": "1.7",
  "metadata": {
    "timestamp": "2026-08-12T10:00:00Z",
    "component": {
      "type": "application",
      "name": "osm-diffs",
      "version": "0.6.10"
    }
  },
  "components": [
    {
      "type": "data",
      "bom-ref": "osm-planet-2026-07-27T15:16:41Z",
      "name": "openstreetmap-planet",
      "properties": [
        {"name": "osm:replication_timestamp", "value": "2026-07-27T15:16:41Z"},
        {"name": "osm:source", "value": null},
        {"name": "osm:writing_program", "value": "planet-dump-ng"}
      ]
    },
    {
      "type": "data",
      "bom-ref": "alltheplaces-2026-03-04-15-16-17",
      "name": "alltheplaces",
      "properties": [
        {"name": "alltheplaces:run_id", "value": "2026-03-04-15-16-17"},
        {"name": "alltheplaces:start_time", "value": "2026-03-04T15:16:17Z"}
      ]
    }
  ]
}

Concretely:

1. Add a metadata.component describing the osm-diffs pipeline itself (name, CARGO_PKG_VERSION, supplier) -- this alone resolves Embed the release version into the pipeline's output data, not just the log #588.
2. Add one type: "data" component per input source actually consumed for a given output file. The underlying attributes for each source are gathered and logged first, one source at a time, ahead of this issue's own embedding work:
   • OpenStreetMap planet dump: done, see Expose OSMHeader provenance metadata from BlobReader #634 (PbfHeader, logged in import_osm).
   • AllThePlaces dump: see Capture and log AllThePlaces run provenance metadata #639.
3. Serialize the document and set it as Parquet key-value metadata (WriterProperties::builder().set_key_value_metadata(...) in src/places/writer.rs, currently unset) when writing each output Parquet file.
4. Decide on a read-back/inspection story for consumers (e.g. pointing people at parquet-tools meta, or a small osm-diffs inspect <file> helper) -- can be a follow-up, not a blocker for the first cut.
5. Add a test asserting the embedded BOM round-trips (e.g. a golden-file test against a small fixture).

Related

• Expose OSMHeader provenance metadata from BlobReader #634 -- the OSM piece (parsing + logging the underlying attributes), merged.
• Capture and log AllThePlaces run provenance metadata #639 -- the AllThePlaces piece (parsing + logging the underlying attributes).
• Embed the release version into the pipeline's output data, not just the log #588 -- pipeline-version stamping; folded into step 1 above.
• CI: Attest build provenance #51 -- CI build-provenance attestation (SLSA-style, about how the binary was built). Different layer from this issue, which is about the lineage of the data a given output file was built from; worth keeping the distinction clear so the two don't get conflated.
• Clean up SBOM generation #540 / Add id-tagging-schema to SBOM #487 -- existing code-SBOM tooling (scripts/sbom/), the precedent for this repo's CycloneDX conventions.

Priority

Not urgent. Filing now so the plan doesn't get lost; #634 and #639 (the prerequisite logging steps) can land independently of this issue's actual embedding work.