Skip to content

geo-agent-training: 'Historical logs' points at nrp:logs-wetlands — dead bucket, wrong credential; open-llm-proxy's copy is already fixed #4

Description

@cboettig

geo-agent-training/SKILL.md line 108, under Historical logs (S3 backup):

rclone copy nrp:logs-wetlands/ ./logs

Three problems, in increasing order of importance.

1. The bucket is dead

logs-wetlands was filled by an llm-proxy-log-backup CronJob that dumped proxy pod stdout nightly. It selects pods by app=llm-proxy, but the deployment has been open-llm-proxy (pod label app=open-llm-proxy) since 2026-03-13 — so it matched zero pods and captured nothing after 2026-04-18, while exiting 0 and reporting success every night. Suspended 2026-08-16; tracked at boettiger-lab/geo-agent-ops#120.

So this command returns 58 MiB frozen at 2026-04-18, presented to the agent as "historical logs" with no indication it stops there. An agent doing log analysis gets a silently truncated picture — worse than an error, because it looks like data.

2. It uses the broad NRP credential

The nrp: remote is general NRP Ceph access — read/write/delete across every bucket. A skill whose whole job is reading logs shouldn't hand out delete on everything. That is the subject of boettiger-lab/open-llm-proxy#113, and a scoped read-only single-bucket credential now exists (boettiger-lab/geo-agent-ops#117, logs-open-llm-proxy-reader, provably unable to write or delete anything).

3. This copy has drifted from the one in open-llm-proxy

boettiger-lab/open-llm-proxy vendors this skill at .claude/skills/geo-agent-training/SKILL.md, and that copy is already correct — it uses ./sync-logs.sh against logs-open-llm-proxy, and its CHANGELOG records the fix ("reached for rclone copy nrp:logs-wetlands/ — the wrong bucket, via the broad NRP…"). The canonical copy here is the stale one, last touched 2026-04-01.

Other staleness in the same file suggests this copy has simply been left behind: it lists fixes as "tracked in" boettiger-lab/open-llm-proxy#1 and #2, both of which are closed (request_id correlation, and origin in response logs — the very limitations the section warns about).

Suggested fix

Sync this copy from open-llm-proxy's, which is current, rather than patching line 108 in isolation — and then decide which of the two is canonical, because a vendored copy that leads the upstream will drift again. The upstream-vs-vendored question is the real one here; the dead bucket is just how it surfaced.

If the 2026-01-08 → 2026-04-18 history is worth keeping at all, that decision is in boettiger-lab/geo-agent-ops#120 — but it shouldn't be reached for via nrp: regardless.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions