Skip to content

Retire catalog/sync/** and the backup/mirror prose in AGENTS.md — ownership moved to geo-agent-ops (supersedes #536) #550

Description

@cboettig

Summary

catalog/sync/** (116 files) and the backup/mirror prose in AGENTS.md are superseded copies of a system this repo no longer owns. Ownership of both sync tiers moved to geo-agent-ops on 2026-07-31; the originals here were deliberately left in place, and removal was never assigned to anyone. They are still being edited — as recently as ca32b35 (2026-08-16) — which means real licensing work is landing in files the live system stopped reading.

Retire them, and reduce this repo's Step 7 obligation to the one thing that is genuinely ours: an accurate STAC license on each collection.

Why they're superseded

geo-agent-ops #66 (closed 2026-07-31) relocated the generators and the manifests:

tier relocated in scope source after the move
minio-sync geo-agent-ops #88 derived live from the NRP catalog (option (b))
source-sync geo-agent-ops #93 curated, in geo-agent-ops scripts/check-source-scope.py

Plus the 59 sync-public-*.yaml and 41 source-sync-*.yaml backfill jobs, replaced there by two templates + scripts/run-backfill.sh.

#66 closed with the originals here suspended, not deleted"removal is owner-only, so the plan deliberately does not depend on it." That was the right call for that issue; it just means nothing tracks the cleanup. geo-agent-ops #21 has no item for it, and #536 covers one file.

What is now actively wrong in AGENTS.md

Step 7 (AGENTS.md:784-840) gives two registration instructions, both stale, in different ways:

  1. MinIO (:806) — "add the name to the BUCKETS array in catalog/sync/minio/gen-minio-sync.sh". Scope is derived from the catalog now, so this edits a file nothing reads. Effect is benign (derivation picks the bucket up regardless), but it manufactures a false belief that registration happened.
  2. source.coop (:816-824) — "record the verdict in license-inventory.md, then realize it in gen-source-sync.sh (REPOS / EXCLUDES)". The live curated scope is in geo-agent-ops. This direction matters more: an exclude recorded only here never reaches the real scope. geo-agent-ops PR cgls-lc100 hex parquet aggregates integer class codes by mean — should use mode #110's auditor independently catches over-publication, so we are not exposed — but the instruction points at the wrong file.

The hard-boundary section (AGENTS.md:156-198) also carries backup operational detail — --max-delete circuit-breaker behavior, cron schedules, backup-dir soft-delete semantics, the rclone-backup secret, the two-failure-domain analysis. None of it is needed to build a dataset, and it is precisely the material that lets an agent here form opinions about a system it is walled off from.

Evidence this is causing live divergence, not just clutter

The two most recent commits touching these files:

ca32b35 (2026-08-16)  fix: ca-climate-zones now MIT — drop source.coop exclude, flip HOLD→OK (#537)
d9ce294               feat: add ca30x30 to the source.coop mirror scope (11 OK, 1 HOLD) (#533)

Between them they did careful per-collection licence research for 12 public-ca30x30 collections (TNC/CDFW ds1197 use constraints, Kling et al. 2018 terms, CAL FIRE FVEG confirmed CC-BY on data.ca.gov, climclust's newly-added MIT LICENSE) and concluded: ca30x30 in REPOS, no excludes, mirror in full.

The live curated scope disagrees, because it never heard about it — geo-agent-ops scripts/check-source-scope.py:

HOLD = {
    ...
    "ca30x30": "partial clearance — only a few collections licence-cleared; rest uncatalogued",
}

So public-ca30x30 is still held out of source.coop on a rationale that #533 directly refutes. The research is the valuable part and it is stranded in a dead file.

Pre-condition: hand the research over before deleting

Deleting license-inventory.md without transferring its findings would destroy sourced licence work. Before removal:

Ask

  • Land the reconciliation above (owner + geo-agent-ops).
  • git rm -r catalog/sync/ (116 files: 2 generators, rewrite-stac-hrefs.py, license-inventory.md, 105 manifests in catalog/sync/k8s/).
  • Rewrite AGENTS.md Step 7 to: register nothing. Backups and mirrors are geo-agent-ops'. This repo's obligation is a correct SPDX license (plus a licence link) on each STAC collection — the advisory input the ops-side auditor reads. Keep the "you never execute backup jobs" boundary; drop the how-it-works.
  • Trim the operational backup detail from AGENTS.md:156-198 down to the credential boundary itself (which secrets exist in geo-workflows, and that MinIO is the sole home for class-3 private data — that part is build-relevant).
  • Close Retire license-inventory.md — redistribution decisions have moved to geo-agent-ops #536 as folded into this (its license-inventory.md retirement is one line of the checklist above).

Why it's worth doing rather than tolerating

The firewall built in geo-agent-ops #21 is about authority — credentials, execution, git write — and it holds. But documentation is an agent's context, so leaving a working model of the backup tier in AGENTS.md means agents here keep reasoning about backups even though they can't touch them. Concretely, reviewing #547 (which purged a superseded 12.6 GiB NRP prefix) I derived a "the weekly wdpa sync will trip --max-delete 150" finding entirely from these leftover files, and raised it as a merge blocker. It was out of scope and redundant: geo-agent-ops #96 landed backup-watch on 2026-08-13, which surfaces a red run within a day and treats a --max-delete trip as a decision on that side. The receiving system already detects the consequence and needs nothing from this repo — which is the design working, and the reason the knowledge should not be here.

Refs

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions