BuildStream builds queue indefinitely when the usb4-link node label is absent

[castrojo]

Problem

docs/skills/cluster-tooling/buildstream.md states a hard admission rule:

When the ghost<->exo-0 USB4 link is down (see RUNBOOK), all cross-node traffic falls back to 2.5GbE, but no BuildStream build may run. Repair the link and wait for fresh lab.projectbluefin.io/usb4-link=up observations on both nodes before submitting or retrying.

Neither node currently carries a usb4-link label at all:

$ kubectl get nodes --show-labels | grep -o 'usb4-link=[a-z]*' | sort -u
(no output)

$ kubectl get nodes
exo-0   Ready
ghost   Ready

Symptom

A build does not get rejected — it queues forever. Observed on an fsdk-containers build: an action completed Uploading input root, then sat in Waiting for the remote build to complete for ~35 minutes while both workers idled at 7m / 14m CPU. Nothing was compiling.

Earlier elements in the same run had built fine (botocore 8m19s, boto3, s3transfer), so this is not a dead grid — it changed state mid-run.

worker-fsgmc   7m    412Mi
worker-n8z6v   14m   850Mi

All pods Running; scheduler on exo-0, workers on ghost and exo-0.

Why this is worth fixing

The documented contract is "no build may run", but the observed behaviour is "builds hang indefinitely". From a caller's perspective those are very different:

• A rejection is actionable in seconds.
• A silent queue looks exactly like a slow build, so callers wait, retry, and eventually start suspecting their own elements. That is where the time goes.

It also means the admission rule is effectively unenforced: nothing stops submission, and there is no signal saying why the work is not progressing.

Expected outcome

Either the label is published so the documented gate can be enforced, or the gate fails closed with a clear error. Concretely, one of:

1. Whatever publishes lab.projectbluefin.io/usb4-link is restored (it appears to not be running — there is no label in either state, up or down).
2. Submission is rejected with a message naming the missing/❌ label, instead of queueing.
3. If the label was retired, the skill doc is corrected so agents stop treating it as the admission contract.

Acceptance criteria

A build submitted while the USB4 link is unavailable either does not start, or fails with an explicit reason — it does not sit in the scheduler queue with idle workers.

Related

• BuildBarn worker hostPath file pool serves corrupted content across builds #637 — worker file pool corruption (separate issue, same grid)