Investigate Egress Billing For Prod Forge Nodes

[hannahhoward]

The production admin dashboard shows no egress except for our internal nodes for January, and none where the client is eng@fil.org. This is surprising.

We need to investigate why this is happening and determine if we have work to fix egress tracking on prod nodes.

[volmedo]

a possibility that could explain what the dashboard shows is that these metrics are delayed due to nodes batching space/content/retrieve receipts before sending them for tracking. By default, piri nodes are configured to do 100MiB batches, and that's a lot of receipts. I think it's likely that no node collected enough receipts only from some tests.

The reason we see egress only on our nodes seem to imply that our nodes are configured with smaller batch sizes. The amount of egress shown is low, which supports this hypothesis, because a batch of 100MiB worth of receipts is likely to represent a good amount of content being egressed (unless users are retrieving many tiny files, and using a space/content/retrieve invocation for each of them).

The 100MiB default batch size was a bit of a random choice and we can lower it down to increase the real-timeness of egress numbers at the cost of more load in the nodes and the etracker.

[volmedo]

I confirmed that our nodes are configured to use 10KiB batches, which explains why they are reporting egress while other production nodes aren't, and that egress amounts are small.

I can only see space/egress/track invocations from our nodes in Jan 2026 in the etracker's table, i.e. no provider nodes requested egress tracking during that period. Track invocations are concentrated on Jan 9, 15, 16, 20 and 21.

[volmedo]

all in all, this seems to be working as expected. We'd see the same numbers if nodes were failing at some point in the egress tracking flow, so the next step would be to ask them to check their nodes' logs to see if there are any related errors.

[frrist]

If a Piri node was offline during a consolidation run is there any chance we could incorrectly mark its batch as consolidated? As outlined here: #11

[volmedo]

If a Piri node was offline during a consolidation run is there any chance we could incorrectly mark its batch as consolidated? As outlined here: #11

yes, that would be covered by the second bullet point in that issue: "Decide whether the egress record should be marked as processed. Depending on the specific error, we might want to keep the record as not processed so that we can retry on a subsequent consolidation cycle."

Currently, errors are logged but don't prevent the batch from being marked as processed, which is obviously not great.