Expose OSMHeader provenance metadata from BlobReader - #634
Merged
Conversation
Parses the fields we need out of the PBF HeaderBlock -- which neither osm_pbf_iter nor protobuf_iter supports -- by hand: writingprogram, source, and osmosis_replication_timestamp. BlobReader::open() now requires a valid replication timestamp to be present (it always is in practice for the planet dumps we consume) and exposes the parsed metadata via BlobReader::header(). replication_sequence_number is omitted: planet-dump-ng, the generator behind the actual planet dumps we process, never sets it. Also logs the metadata at INFO level when the planet file is opened, so provenance shows up in pipeline.log even before we start embedding it into our output files. Note for reviewers: osmosis_replication_timestamp is a plain int64, not sint64, so it is not zigzag-encoded on the wire -- decoding it via protobuf_iter's i64::from(ParseValue) (which always zigzag-decodes) would silently corrupt the value. Decode via u64::from instead, which returns the raw varint.
brawer
commented
Aug 12, 2026
This was referenced Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
BlobReadernow parses theOSMHeaderblock'swritingprogram,source, andosmosis_replication_timestampfields by hand (neitherosm_pbf_iternorprotobuf_itersupport that message) and exposes them via a newBlobReader::header() -> &PbfHeader.replication_timestampis required:BlobReader::open()now errors if the header block is missing or has no timestamp. In practice this never happens for the planet dumps we process.replication_sequence_numberis intentionally not exposed —planet-dump-ng, the tool behind the actualplanet.openstreetmap.orgdumps, never sets it in the header.import_osmlogs the parsed metadata at INFO level (structured, vialog's key-value API) right after opening the planet file, so provenance is visible inpipeline.logahead of a future change to embed it into our output files.Why
We want to eventually stamp our output files with the provenance of the input planet data (replication timestamp etc.). This is the first step: get
BlobReaderto expose it and log it. Embedding it into output files is a follow-up.Testing
cargo test --lib pipeline::osm::testsandcargo test --test integration_testpass.cargo fmt --checkandcargo clippy --lib -- -D warningsare clean.zugerland.osm.pbftest fixture (not the real planet dump):{"fields":{"replication_timestamp":"2026-01-27T08:11:02Z","source":null,"writing_program":"osmx"},"level":"INFO","message":"opened OpenStreetMap planet file", ...}workdir/osm-planet.pbfby hand-decoding the header block with a throwaway script, to confirmosmosis_replication_sequence_numberreally is absent from real planet dumps and not just from the small test fixture.🤖 Generated with Claude Code