nllb: Envoy health check unhealthy_threshold: 5 causes 25s detection lag on controller failure

[ncopa]

Problem

The Envoy configuration for the NLLB apiserver and konnectivity clusters uses:

health_checks:
- tcp_health_check: {}
  timeout: 1s
  interval: 5s
  healthy_threshold: 3
  unhealthy_threshold: 5

interval: 5s × unhealthy_threshold: 5 means Envoy takes 25 seconds after a controller stops before it removes it from rotation. During those 25 seconds, konnectivity reconnection attempts from workers are routed to the dead controller and fail, delaying cluster recovery in HA deployments.

There is already a FIXME comment at the relevant location noting the health check needs improvement:

// pkg/component/worker/nllb/envoy.go:375
// FIXME: Better use a proper HTTP based health check, but this needs certs and stuff...
- tcp_health_check: {}

This delay was also identified as a contributing factor to the check-nllb-ipv6 / check-nllb-traefik-ipv6 test flakiness (see #7742).

Desired solution

Lower unhealthy_threshold from 5 to 2 for both clusters (apiserver and konnectivity) in pkg/component/worker/nllb/envoy.go:

# Before
unhealthy_threshold: 5   # 5 × 5s = 25s to declare a controller dead

# After
unhealthy_threshold: 2   # 2 × 5s = 10s — still conservative, avoids flaps

This cuts the dead-controller detection window from 25s to 10s without risking false positives on transient network blips.

[twz123]

konnectivity reconnection attempts from workers are routed to the dead controller and fail, delaying cluster recovery in HA deployments.

AFAIK, the konnectivity agent just brute-forces connection attempts until it has the desired amount of server connections, and envoy is either using round-robin or random LB strategies, so I don't understand yet why this is a problem? I mean, yes, konnectivity will log quite some errors, but why does it try to connect in the first place? I think we should investigate why it thinks it has to connect to yet another konnectivity server? I don't understand yet why taking a server out of LB earlier is making the described situation any better. The agent will just try to hammer the remaining konnectivity servers with connection attempts, then?