Skip to content

Stop custom multi-buy backoff from dropping packets for minutes - #343

Open
macpie wants to merge 2 commits into
mainfrom
macpie/hpr-backoff-packet-drops-c6bb79
Open

macpie wants to merge 2 commits into
mainfrom
macpie/hpr-backoff-packet-drops-c6bb79

Conversation

@macpie

@macpie macpie commented Sep 23, 2026

Copy link
Copy Markdown
Member

Every request that was in flight when a custom multi-buy channel died called inc_backoff separately, so one dropped connection doubled the delay once per failed request. Eight concurrent failures walked the window from 1s to over two minutes, and on a route with fail_on_unavailable set that meant every packet was dropped for the whole time. None of it was visible either: the backoff short-circuit and the cold-channel path emitted no metrics, so a route dropping everything looked the same as a route with no traffic.

  • Escalate the backoff once per window instead of once per failed request.
  • Cap the window at 60s for routes that fail closed. Dropping every packet is a heavy penalty and shouldn't ride the 5 minute ceiling meant for routes that fall back to local counting.
  • Let a single probe through when a window expires and hold the rest back until it reports, so a service that is still down costs one call rather than one per packet. The hold expires on its own, so a probe that never answers can't wedge the route.
  • Count every decision in hpr_multi_buy_decision_counter, including the drops that were previously silent.
  • Stop the grpc channel and clear its backoff entry when a route is removed or repointed at a different host, instead of leaking both for the life of the node.

Every request that was in flight when a custom multi-buy channel died called
inc_backoff separately, so one dropped connection doubled the delay once per
failed request. Eight concurrent failures walked the window from 1s to over
two minutes, and on a route with fail_on_unavailable set that meant every
packet was dropped for the whole time. None of it was visible either: the
backoff short-circuit and the cold-channel path emitted no metrics, so a route
dropping everything looked the same as a route with no traffic.

- Escalate the backoff once per window instead of once per failed request.
- Cap the window at 60s for routes that fail closed. Dropping every packet is
  a heavy penalty and shouldn't ride the 5 minute ceiling meant for routes
  that fall back to local counting.
- Let a single probe through when a window expires and hold the rest back
  until it reports, so a service that is still down costs one call rather than
  one per packet. The hold expires on its own, so a probe that never answers
  can't wedge the route.
- Count every decision in hpr_multi_buy_decision_counter, including the drops
  that were previously silent.
- Stop the grpc channel and clear its backoff entry when a route is removed or
  repointed at a different host, instead of leaking both for the life of the
  node.
helium/enacl declared the NIF unload hook with the upgrade signature, which
older compilers only warned about. The base image ships gcc 14, where
-Wincompatible-pointer-types is an error, so compiling the C sources fails
outright. CI got away with it until now only because the _build cache always
carried a prebuilt enacl_nif.so; once that cache expired, the first
from-scratch build of the test profile broke.

libp2p_crypto already asks for enacl master, which has carried the fix since
helium/enacl#2, but our lock still pinned a commit from before it. rebar3
refuses to upgrade a transitive dep, so promote enacl to a top-level dep and
relock it. The corrected unload signature is the only difference between the
two commits.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants