feat: support LoadBalancer service backed by vpc nat gateway nft shar… - #7047
Draft
zbb88888 wants to merge 2 commits into
Draft
feat: support LoadBalancer service backed by vpc nat gateway nft shar…#7047zbb88888 wants to merge 2 commits into
zbb88888 wants to merge 2 commits into
Conversation
Coverage Report for CI Build 31781692423Coverage decreased (-0.03%) to 32.447%Details
Uncovered Changes
Coverage RegressionsNo coverage regressions found. Coverage Stats
💛 - Coveralls |
Collaborator
Author
|
核心结论：数据面负载均衡的核心逻辑与 kube-proxy 的 nftables 后端基本一致，但覆盖范围（Service 语义面）明显更窄。 一致的部分（核心 LB 机制）
亲和的 nft 规则我用 mock 逐条对照过 kube-proxy 的 有差异 / 未覆盖的部分（范围更窄）
一句话总结
|
zbb88888
force-pushed
the
support-vpc-lb-based-nft-dnat
branch
from
July 20, 2026 08:46
0fc79ed to
795b409
Compare
zbb88888
force-pushed
the
support-vpc-lb-based-nft-dnat
branch
4 times, most recently
from
July 30, 2026 07:24
a7ea49c to
20cf8fe
Compare
zbb88888
force-pushed
the
support-vpc-lb-based-nft-dnat
branch
2 times, most recently
from
August 7, 2026 05:46
fe70a24 to
f8b14b4
Compare
…e DNAT
Make a vpc-nat-gw act as an in-VPC, kube-proxy-like load balancer for
LoadBalancer type Services, so a Service whose backends live in a custom
VPC (unreachable by the host kube-proxy) gets a working EXTERNAL-IP.
Enabled by --enable-nftable-lb-svc (requires --enable-lb). A Service is
handled when it is type=LoadBalancer and carries ovn.kubernetes.io/eip;
the gateway is derived from the EIP's natGwDp. The controller watches the
Service and its EndpointSlices and programs one share type
IptablesDnatRule per (servicePort, ready backend); the nftables map-based
share DNAT (numgen random) then balances new connections and pins them
per-connection via conntrack.
Design
------
One EIP, two forwarding planes, each targeting the backend NIC reachable
in its own domain (the recommended backend is dual-NIC and listens on
0.0.0.0):
* external and custom-VPC-internal traffic reaches the EIP through the
NAT gateway (BGP/macvlan ARP resolves the EIP to the gateway pod, so
it never enters the host stack) and is DNAT'd to the backend's
gateway-VPC NIC;
* default-VPC-internal / host traffic reaches the EIP (published as the
Service ingress IP) through kube-proxy's own ipvs0 local route and is
balanced to the backend's primary/default-VPC endpoint IP.
The two planes are mutually exclusive per packet (L2-to-pod-MAC vs host
L3 routing), so they coexist without conflict and kube-proxy is left
untouched.
Backend IP selection: each ready endpoint is resolved to the NIC that
sits in the gateway's VPC (via the pod's kube-ovn providers/subnets), not
the k8s endpoint (primary) IP the gateway cannot route to. A single-NIC
backend already in the gateway VPC matches on its only NIC (unchanged
behavior); a backend with kube-ovn NICs but none in the gateway VPC is
skipped with a one-shot Warning event to avoid a black-hole share DNAT;
when several NICs share the gateway VPC the lowest IPv4 is chosen
deterministically (dual-NIC covers ~99% of cases). Resolution reuses the
EndpointSlice informer for readiness and is memoized per pod.
Ownership & lifecycle: the cluster-scoped IptablesDnatRule cannot use an
OwnerReference to a namespaced Service, so generated rules are labeled
with the owning Service and reconciled (create/delete) on Service and
EndpointSlice changes; Spec drift (affinity, timeout, EIP) triggers
delete+recreate because share rules are immutable once Ready.
Conflict handling: a share identity (eip:port:proto) is a single nft map
and can be programmed by only one owner. The deterministic winner is
derived from the Service objects referencing the same EIP (stable intent,
race-free regardless of reconcile order), looked up via a service
informer index keyed by the eip annotation; a manually-created share rule
always wins. Losers back off with an NftableLbSvcConflict event and
requeue to take over once the identity is released. The validating
webhook rejects cross-owner same-identity share rules (and enforces a
consistent session-affinity across an identity), skipping terminating
rules so a recreate is not blocked.
ClientIP session affinity: add SessionAffinity/SessionAffinityTimeoutSeconds
to IptablesDnatRule (share only) and implement the kube-proxy nftables
affinity pattern in nat-gateway.sh (per-backend timeout sets + source-IP
goto layered on the same numgen random dispatch). The stateless (none)
path is unchanged.
Advantages
----------
* data plane isolated from the host: kube-proxy needs no change and no
black-hole; the same EIP serves external, custom-VPC and default-VPC
clients, matching k8s LoadBalancer semantics;
* reuses the k8s EndpointSlice informer for backend readiness/rollout;
* deterministic, race-free conflict resolution from stable Service
intent plus an O(matched) informer index (no full-store scans, no
mutation of user Service objects);
* backward-compatible wire format: nat-gateway.sh accepts both the
legacy 4-field and the new 6-field map-add form and keeps identity
chain naming unchanged for rolling upgrades.
Scope: aligns with kube-proxy's Cluster traffic policy only (all Ready
endpoints; no ExternalTrafficPolicy/InternalTrafficPolicy=Local,
topology, or terminating-endpoint fallback); IPv4 and tcp/udp only.
Manifests: wire ENABLE_NFTABLE_LB_SVC into install.sh and both charts;
enable it for the iptables-vpc-nat-gw e2e job.
Tests: unit tests for rule/affinity generation, backend NIC selection,
conflict resolution and the service-by-eip index; e2e for lifecycle,
ingress IP, dual-NIC backend selection, service/manual conflict, ClientIP
affinity stickiness and data-path distribution.
Signed-off-by: zbb88888 <jmdxjsjgcxy@gmail.com>
zbb88888
force-pushed
the
support-vpc-lb-based-nft-dnat
branch
from
August 7, 2026 07:45
f8b14b4 to
560bb7a
Compare
Collaborator
Author
|
完整测试报告已生成并保存到 /root/myskills/ai-tests/kubeovn/e2e-vpc-lb/TEST_REPORT.md。以下是完整报告内容： ──────────────────────────────────────────────────────────────────────────────── kube-ovn nftable-lb-svc E2E 测试报告 │ 测试对象 PR：feat: support LoadBalancer service backed by vpc nat gateway nft share DNAT ────────────────────────────────────────────────────────────────────────────────
运行记录：run 1 ✅ 全绿 → run 2 ❌（环境故障：apiserver↔etcd 连接超时级联重启，非用例问题）→ run 3 ✅ 全绿 → run 4 ✅ 全绿（本报告引用其完整证据）
数据面架构（实测确认）：
生成的规则携带归属标签 ovn.kubernetes.io/nftable-lb-svc-ns/name（cluster-scoped 规则替代 OwnerReference），type: share，status.ready=true。✅ PASS
核心：控制器按 pod 的 provider/subnet 解析出网关 VPC 侧 NIC，而非 kube-proxy 可见的默认 VPC endpoint。✅ PASS
✅ PASS
手工规则（owner 为空）永远优先，特性不覆盖手管规则。✅ PASS
✅ PASS（粘性 100%）
数据路径：客户端(eg-subnet) → OVN 静态路由 → gw eth0(打标) → nft DNAT(numgen) → 后端 → 回包经 HAIRPIN_SNAT + conntrack 逆变换回客户端。✅ PASS
✅ PR 的 6 个 e2e 用例在真实集群全部通过，三轮完整套件复跑全绿（27 个步骤级 PASS），验证了规则自动生成/生命周期、nft numgen 负载均衡（15/15）与后端增减重建（mod 2→1→移除）、双网卡后端 net1 ──────────────────────────────────────────────────────────────────────────────── 报告与脚本均维护在 /root/myskills/ai-tests/kubeovn/e2e-vpc-lb/：
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
What type of this PR
Description
Make a
vpc-nat-gwact as an in-VPC, kube-proxy-like load balancer fortype=LoadBalancerServices, so a Service whose backends live in a custom VPC(unreachable by the host kube-proxy) gets a working
EXTERNAL-IP.Enabled by
--enable-nftable-lb-svc(requires--enable-lb). A Service is handled when it istype=LoadBalancerand carriesovn.kubernetes.io/eip; the gateway is derived from the EIP'snatGwDp. The controller watches the Service and its EndpointSlices and programs one sharetype
IptablesDnatRuleper(servicePort, ready backend). The nftables map-based share DNAT(
numgen random, added in #6858) then balances new connections and pins them per-connectionvia conntrack.
Design — one EIP, two forwarding planes
The same EIP is reachable through two planes, each targeting the backend NIC reachable in its
own domain. The recommended backend is dual-NIC and listens on
0.0.0.0:kube-ipvs0local routeThe two planes are mutually exclusive per packet (L2-to-pod-MAC vs host L3 routing;
arp_ignore=1makes only the gateway pod answer ARP for the EIP), so they coexist withoutconflict and kube-proxy is left untouched.
Backend IP selection. Each ready endpoint is resolved to the NIC that sits in the
gateway's VPC (via the pod's kube-ovn providers/subnets), not the k8s endpoint (primary) IP the
gateway cannot route to:
event, to avoid a black-hole share DNAT;
Resolution reuses the EndpointSlice informer for readiness and is memoized per pod.
Ownership & lifecycle. The cluster-scoped
IptablesDnatRulecannot use an OwnerReference toa namespaced Service, so generated rules are labeled with the owning Service and reconciled on
Service/EndpointSlice changes; Spec drift (affinity, timeout, EIP) triggers delete+recreate
because share rules are immutable once Ready.
Conflict handling. A share identity
(eip:port:proto)is a single nft map programmable byonly one owner. The deterministic winner is derived from the Service objects referencing the
same EIP (stable intent, race-free regardless of reconcile order), looked up via a service
informer index keyed by the eip annotation; a manually-created share rule always wins. Losers
back off with an
NftableLbSvcConflictevent and requeue. The validating webhook rejectscross-owner same-identity share rules and enforces consistent session-affinity across an
identity, skipping terminating rules so a recreate is not blocked.
ClientIP session affinity. Adds
SessionAffinity/SessionAffinityTimeoutSecondstoIptablesDnatRule(share only) and implements the kube-proxy nftables affinity pattern innat-gateway.sh(per-backend timeout sets + source-IPgoto, layered on the samenumgen randomdispatch). The stateless (none) path is unchanged.Advantages
same EIP serves external, custom-VPC and default-VPC clients (k8s LoadBalancer semantics).
checking.
O(matched)informer index (no full-store scans, no mutation of user Service objects).
nat-gateway.shaccepts both the legacy 4-field and thenew 6-field map-add form and keeps identity chain naming unchanged for rolling upgrades.
Implementation
pkg/apis/kubeovn/v1/iptables-dnat-rule.go):SessionAffinity(enum""/ClientIP)and
SessionAffinityTimeoutSeconds(0–86400, default 10800).pkg/controller/nftable_lb_svc.go, new): reconcile, backend NIC resolver,conflict resolution, ingress-IP publish/clear;
indexers.goservice-by-eip index;controller.go/config.go/service.go/endpoint_slice.gowiring and enqueue hooks.pkg/controller/nft_dnat.go,vpc_nat_gw_nat.go,dist/images/vpcnatgateway/nat-gateway.sh): thread affinity into the share DNAT map-add/cleanup.pkg/webhook/vpc_nat_gateway.go): affinity + cross-owner identity validation.pkg/util/vpc_nat_gateway.go): single decoderNftableLbSvcOwnerKeyused byboth controller and webhook.
ENABLE_NFTABLE_LB_SVCwired intoinstall.shand both charts; enabled for theiptables-vpc-nat-gwe2e job.Scope: aligns with kube-proxy's Cluster traffic policy only (all Ready endpoints; no
ExternalTrafficPolicy/InternalTrafficPolicy=Local, topology, or terminating-endpointfallback); IPv4 and tcp/udp only.
Tests
index.
affinity stickiness, and data-path distribution.
flowchart LR ExtC["External client"] CvpcC["Custom-VPC pod"] DvpcC["Default-VPC pod / host"] EIP(["EIP<br/>(Service EXTERNAL-IP)"]) subgraph GW["Plane A · vpc-nat-gw (nft share DNAT)"] GWpod["gateway pod net1 = EIP<br/>numgen random + ClientIP affinity"] end subgraph KP["Plane B · kube-proxy (untouched)"] IPVS["kube-ipvs0 (EIP local route)<br/>ipvs load balance"] end subgraph POD["Dual-NIC backend pod · listens 0.0.0.0"] NIC2["gateway-VPC NIC"] NIC1["primary / default-VPC NIC"] end ExtC -- "BGP/macvlan · ARP→gw pod MAC<br/>(bypasses host stack)" --> EIP CvpcC -- "route to gateway<br/>(hairpin SNAT)" --> EIP EIP --> GWpod GWpod -- "DNAT → gateway-VPC NIC IP" --> NIC2 DvpcC -- "host L3 → local table" --> IPVS IPVS -- "DNAT → primary endpoint IP" --> NIC1 classDef plane fill:#eef,stroke:#88a; classDef pod fill:#efe,stroke:#8a8; class GW,KP plane; class POD pod;要点(图与设计一致):
arp_ignore=1保证只有网关 pod 应答 EIP 的 ARP。kube-ipvs0本地路由 → 后端主网卡(kube-proxy 零改动)。Pull Request
What type of this PR
Examples of user facing changes:
Which issue(s) this PR fixes
Fixes #(issue-number)