Unexplained volume of stamp purchases on the staging node

[crtahlin]

Open question

A staging Bee node accumulated 82 postage stamps against a configured reserve of 5, spending roughly 8.9 BZZ (balance 10 -> 0.80). #265 fixed one mechanism that caused this — the pool bought a full reserve whenever it could not read the node — but that mechanism accounts for five stamps per affected restart, not seventy.

What the evidence shows

Reconstructing purchase times from the remaining stamps' TTLs:

 1h ago: 51 stamps
 2h ago: 25
14h ago: 5
15h ago: 1

76 of 82 originate in a roughly two-hour window, most of it before that day's first deploy. That does not match one-reserve-per-restart, which would spread purchases across deploys.

Candidate explanations, none confirmed

1. External callers. The gateway had become publicly reachable, and the free tier permits 3 requests/minute with no payment. POST /api/v1/stamps/ and /pool/acquire are both reachable that way, and an acquire removes a stamp from the pool, prompting a replacement purchase. Sustained traffic would produce exactly this shape.
2. A second defect in the replenish path beyond the one Do not replenish the stamp pool when the Bee node cannot be read #265 addressed.
3. Repeated restarts from some source not visible in the surviving logs.

Why it could not be resolved

• The reverse proxy was not recording access logs at the time, so there is no record of inbound requests. (Now fixed — access logging is enabled.)
• The gateway container holding its own request log was replaced, taking the log with it.
• gateway_stamp_purchases_total is a counter that resets on every restart, and no series is currently exported, so the history is not in Prometheus either.

That last point is worth addressing on its own: the metric that exists specifically to answer "how many stamps did we buy and when" could not answer it. A counter that resets on restart, with no recording rule or long-term aggregation, gives no usable history.

Current state

After #265 the node has been stable: 82 stamps and an unchanged balance across six samples over eight minutes. The bleeding has stopped, but the cause of the bulk of those purchases is unidentified, and the same conditions could recur.

Suggested follow-up

• Watch stamp count and wallet balance now that access logs exist; if it recurs, the logs will show whether inbound traffic drives it.
• Consider a hard ceiling on pool purchases per interval, so no defect in the replenish path can spend without bound.
• Make purchase history durable — a recording rule, or a gauge of node-owned stamps (gateway_node_stamps_total, added in Add node-owned stamp metrics; correct misleading gauge descriptions #262) which survives restarts because it is derived from the node rather than counted in-process.

[crtahlin]

Likely cause identified — see #274

#274 describes a defect that fits this far better than the once-per-restart explanation above.

sync_from_bee_node() parses amount with int(stamp_data.get("amount", 0)), which raises on an empty string or null rather than falling back to the default. Bee returns exactly that: all 421 batches from /batches currently carry amount: null. The try wraps the whole loop, so one unparseable record aborts the entire sync.

Before #265, an aborted sync returned 0, check_and_replenish() read that as an empty pool, and it bought a full reserve — on every check interval, not once per restart. With STAMP_POOL_CHECK_INTERVAL_SECONDS=60, a sustained sync failure would buy continuously.

That matches the evidence here in a way the deploy-count theory never did:

• the purchases cluster tightly in a short window rather than spreading across deploys
• the volume is far larger than one reserve per restart could produce
• it requires no external traffic, so the free-tier hypothesis is not needed to explain it

Not yet proven — the logs from that window are gone with the container that held them — but it is a concrete mechanism that produces the observed shape, rather than an unexplained gap.

The spend is already stopped by #265's guard. #274 covers making the sync survive a single bad record, which is what would prevent the underlying condition recurring.

[crtahlin]

The open question is now largely answered, using history that did not exist when this was filed. Metrics are remote-written to Grafana Cloud, so counters survive restarts and the series goes back weeks.

The mechanism

Acquires drive replacement purchases, and purchases track acquires closely. Staging, per day:

           acquires   purchases
08-25           664          3
08-26         3,866         40
08-27            21          3
08-28+         ~4/day        0

Node-owned batches over the same period peaked at 286 on staging — well beyond the 82 recorded here — and 18 on production.

This is candidate explanation 1 from the original report: an acquire removes a batch from the pool and prompts a replacement purchase, so sustained acquire traffic produces sustained buying. It is not a defect in the replenish path; the pool is doing what it is designed to do.

I reproduced it by accident

The 08-26 spike is mine. The provenance-smasher suite ran unbounded against staging that day and issued thousands of acquires; I attributed 154 consumed batches and about 7.7 BZZ to it at the time, and the metrics now put the acquire count at 3,866. That is the same shape as the original incident, from a caller that was not malicious and not even aware it was spending.

Which supports the original reading: the gateway was publicly reachable, the free tier permits 3 requests a minute with no payment, and each successful acquire costs a batch. No attacker is required — ordinary traffic does it.

What is now in place

A hard ceiling on purchases. STAMP_POOL_MAX_PURCHASES_PER_HOUR, default 10, enforced inside _purchase_stamp so every path is covered, counted when Bee accepts the batch rather than when it becomes usable, and reported in /pool/status rather than only logged.

Sizing it against this incident: 76 of the 82 batches were bought inside roughly two hours, about 38 an hour. A ceiling of 10 would have stopped it at roughly 20 instead of 76, and surfaced an error saying why. It does not prevent slow sustained drain, and is not meant to — it bounds the blast radius of anything fast.

A smaller pool. The target is now one medium batch rather than five, so there is less to drain and less to replace.

Ownership. Batches the pool holds are registered to the gateway and refused to direct callers (#312). That closes a related path where a caller could consume the batch\s capacity without acquiring it at all.

What is not addressed

The underlying economics: an unauthenticated free tier that hands out batches the gateway paid for. The rate limit is 3 requests a minute per IP, which over a day is a lot of batches if someone wants them. That is a product decision rather than a defect, and it belongs with #102 (what the free tier should cost and who may call it) rather than here.

Suggested close

The question this issue asks — where did the purchases come from — is answered. The remaining concern is bounded by the ceiling and reduced by the smaller pool. I would close it and let the free-tier economics be argued in #102, but leaving that call to you.