Surfaced while auditing raw/ mirror exclusions (geo-agent-ops#106). Filing as a flag, not a change — this is a data-workflows call.
Two collections publish an asset from under raw/
Found by crawling all 266 STAC documents for asset hrefs containing /raw/:
| collection |
bucket |
asset href |
size |
ghs-pop-2020 |
public-population |
raw/ghs-pop-2020-cog.tif |
11.60 GiB |
landvote |
public-tpl |
landvote/raw/Measures.csv |
0.77 MiB |
The population case is the clear one. raw/ there holds two objects:
10.11 GiB raw/GHS_POP_E2020_GLOBE_R2023A_4326_3ss_V1_0.tif <- genuine upstream source
11.60 GiB raw/ghs-pop-2020-cog.tif <- the PUBLISHED COG
The derived product and the upstream input share a prefix whose name says "upstream input". Everywhere else in the catalog, raw/ means re-downloadable provider source staged for build restart — which is exactly why it is now excluded from the MinIO mirror.
Why it costs something
raw/ is now excluded from the MinIO backup mirror for 40 of 42 buckets. population and tpl are exempt — meaning their raw/ prefixes, including 10.11 GiB of genuinely re-downloadable source, are still mirrored, because the exclusion cannot distinguish the published COG from the input beside it.
The exemption is verified against STAC on every scope build and fails closed if it drifts, so nothing is silently at risk. But it is an exemption encoding a placement inconsistency, and it will outlive anyone's memory of why.
Suggested
Standardise where a COG lives, and move these to it — something like public-population/ghs-pop-2020-cog.tif (or the collection-prefixed layout used elsewhere), leaving raw/ for provider source only. Re-registering the STAC asset href is the other half.
Once the assets no longer resolve under raw/, the exemption retires itself: the verifier will report exempt but no longer publishes from raw/, and the entry gets deleted. That converts a permanent special case into a one-time fix, and recovers the ~10 GiB of genuine source from the mirror as a side effect.
No urgency — nothing is broken and nothing is at risk. Worth doing when the collection is next touched.
Surfaced while auditing
raw/mirror exclusions (geo-agent-ops#106). Filing as a flag, not a change — this is a data-workflows call.Two collections publish an asset from under
raw/Found by crawling all 266 STAC documents for asset hrefs containing
/raw/:ghs-pop-2020public-populationraw/ghs-pop-2020-cog.tiflandvotepublic-tpllandvote/raw/Measures.csvThe
populationcase is the clear one.raw/there holds two objects:The derived product and the upstream input share a prefix whose name says "upstream input". Everywhere else in the catalog,
raw/means re-downloadable provider source staged for build restart — which is exactly why it is now excluded from the MinIO mirror.Why it costs something
raw/is now excluded from the MinIO backup mirror for 40 of 42 buckets.populationandtplare exempt — meaning theirraw/prefixes, including 10.11 GiB of genuinely re-downloadable source, are still mirrored, because the exclusion cannot distinguish the published COG from the input beside it.The exemption is verified against STAC on every scope build and fails closed if it drifts, so nothing is silently at risk. But it is an exemption encoding a placement inconsistency, and it will outlive anyone's memory of why.
Suggested
Standardise where a COG lives, and move these to it — something like
public-population/ghs-pop-2020-cog.tif(or the collection-prefixed layout used elsewhere), leavingraw/for provider source only. Re-registering the STAC asset href is the other half.Once the assets no longer resolve under
raw/, the exemption retires itself: the verifier will reportexempt but no longer publishes from raw/, and the entry gets deleted. That converts a permanent special case into a one-time fix, and recovers the ~10 GiB of genuine source from the mirror as a side effect.No urgency — nothing is broken and nothing is at risk. Worth doing when the collection is next touched.