Skip to content

Expose OSMHeader provenance metadata from BlobReader - #634

Merged
brawer merged 2 commits into
mainfrom
blob-reader-header-metadata
Aug 12, 2026
Merged

Expose OSMHeader provenance metadata from BlobReader#634
brawer merged 2 commits into
mainfrom
blob-reader-header-metadata

Conversation

@brawer

@brawer brawer commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Summary

  • BlobReader now parses the OSMHeader block's writingprogram, source, and osmosis_replication_timestamp fields by hand (neither osm_pbf_iter nor protobuf_iter support that message) and exposes them via a new BlobReader::header() -> &PbfHeader.
  • replication_timestamp is required: BlobReader::open() now errors if the header block is missing or has no timestamp. In practice this never happens for the planet dumps we process. replication_sequence_number is intentionally not exposed — planet-dump-ng, the tool behind the actual planet.openstreetmap.org dumps, never sets it in the header.
  • import_osm logs the parsed metadata at INFO level (structured, via log's key-value API) right after opening the planet file, so provenance is visible in pipeline.log ahead of a future change to embed it into our output files.

Why

We want to eventually stamp our output files with the provenance of the input planet data (replication timestamp etc.). This is the first step: get BlobReader to expose it and log it. Embedding it into output files is a follow-up.

Testing

  • cargo test --lib pipeline::osm::tests and cargo test --test integration_test pass.
  • cargo fmt --check and cargo clippy --lib -- -D warnings are clean.
  • Manually verified the logged JSON line against the zugerland.osm.pbf test fixture (not the real planet dump):
    {"fields":{"replication_timestamp":"2026-01-27T08:11:02Z","source":null,"writing_program":"osmx"},"level":"INFO","message":"opened OpenStreetMap planet file", ...}
  • Cross-checked the field parsing (tags, wire types, real values) against both the checked-in test fixture and the real planet dump in workdir/osm-planet.pbf by hand-decoding the header block with a throwaway script, to confirm osmosis_replication_sequence_number really is absent from real planet dumps and not just from the small test fixture.

🤖 Generated with Claude Code

Parses the fields we need out of the PBF HeaderBlock -- which neither
osm_pbf_iter nor protobuf_iter supports -- by hand: writingprogram,
source, and osmosis_replication_timestamp. BlobReader::open() now
requires a valid replication timestamp to be present (it always is in
practice for the planet dumps we consume) and exposes the parsed
metadata via BlobReader::header(). replication_sequence_number is
omitted: planet-dump-ng, the generator behind the actual planet dumps
we process, never sets it.

Also logs the metadata at INFO level when the planet file is opened,
so provenance shows up in pipeline.log even before we start embedding
it into our output files.

Note for reviewers: osmosis_replication_timestamp is a plain int64,
not sint64, so it is not zigzag-encoded on the wire -- decoding it via
protobuf_iter's i64::from(ParseValue) (which always zigzag-decodes)
would silently corrupt the value. Decode via u64::from instead, which
returns the raw varint.
Comment thread src/pipeline/osm/mod.rs
Link the upstream planet-dump-ng issues: the actual planet dumps we
process never write osmosis_replication_sequence_number, and it's a
known, still-open gap upstream (issue #16), blocked on a harder
problem (issue #6) rather than intentionally omitted.

@brawer brawer left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant