e2e: use server node IP for Calico eBPF kubernetes endpoint - #10936
e2e: use server node IP for Calico eBPF kubernetes endpoint#10936thomasferrandiz wants to merge 1 commit into
Conversation
Replace hardcoded 'localhost' with the actual node IP in the kubernetesServiceEndpoint host field of the Calico eBPF manifest. The node IP is passed as an argument from the Vagrantfile provisioner. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Codecov Report✅ All modified and coverable lines are covered by tests.
Additional details and impacted files@@ Coverage Diff @@
## master #10936 +/- ##
===========================================
- Coverage 19.84% 8.79% -11.05%
===========================================
Files 50 39 -11
Lines 4133 3059 -1074
===========================================
- Hits 820 269 -551
+ Misses 3262 2776 -486
+ Partials 51 14 -37
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| linuxDataplane: BPF | ||
| kubernetesServiceEndpoint: | ||
| host: localhost" > /var/lib/rancher/rke2/server/manifests/rke2-calico-config.yaml | ||
| host: ${NODE_IP}" > /var/lib/rancher/rke2/server/manifests/rke2-calico-config.yaml |
There was a problem hiding this comment.
Do we really tell people to hardcode the IP of a single node in the chart values? That is terrible, we must not do that. Won't this break the calico for the whole cluster if that one node is down?
Why can't we use localhost:6443? This is guaranteed to be available on both servers and agents - if the node is not a control-plane node, there's a supervisor load-balancer listening on that port that forwards to an apiserver. This is build in to RKE2 and does not rely on kube-proxy.
There was a problem hiding this comment.
We've had reports from customers (SURE-11652 for example) that using localhost doesn't work in that case.
The Calico pod calico-kube-controllers is not using the host network so it doesn't resolve localhost:6643 as the load-balancer endpoint.
That's also why I added the doc about using HAProxy for when users need HA with Calico eBPF.
There was a problem hiding this comment.
Hmm. These are all ugly solutions that we should come up with a better answer for.
Can we modify calico-kube-controllers so that it does run with host network? Or does it need to be in the cluster for some reason?
If running with host network is not desirable for some reason, we should use downward API to inject the IP of the node the pod is running on (status.hostIP), instead of hardcoding a single IP in the values.
There was a problem hiding this comment.
We can't easily modify calico-kube-controllers because it's deployed by the Tigera Operator so we don't control the daemonset configuration.
I'll check if I can find a way to use status.hostIP in the KubernetesEndPoint configmap.
There was a problem hiding this comment.
It looks like we can't do that either without changing the operator itself.
We could always open an issue upstream but there was projectcalico/calico#9141 already.
Their opinion seems to be that it's OK to have a non-redundant connection to the API server because the downtime would be short anyway.
Maybe kube-vip would be a better fit for this than Haproxy?
There was a problem hiding this comment.
This seems like a really poor design in general... either you need to make one of your nodes a single point of failure for the whole cluster, or you need to stand up your own LB in front of the apiserver?
If it would just look at the Kubernetes Endpoints/EndpointSlice instead of the service ClusterIP, it would get the IPs of all the control-plane nodes, and could connect to them directly without having to do anything at all.
There was a problem hiding this comment.
I agree and we could try to submit a PR upstream but it doesn't look like they care.
eBPF is most likely there to check a box compared to Cilium it's not their main data plane.
3 participants
Proposed Changes
The Calico eBPF e2e test manifest hardcoded
localhostas thekubernetesServiceEndpoint.host, which is incorrect when Calico eBPF is configured to bypass kube-proxy and connect to the Kubernetes API server directly. The endpoint should be the actual IP address of the server node so that Calico can reach the API server over the network rather than loopback.This makes the e2e test match our documentation on using Calico with eBPF.
The node's IPv4 address is passed as an argument from the Vagrantfile provisioner to the manifest script, and used in place of
localhost.Types of Changes
Bug fix in e2e test infrastructure.
Verification
Run the Calico eBPF e2e test suite:
Verify that
rke2-calico-config.yamlon the server node containshost: 10.10.10.100instead ofhost: localhost.Testing
Covered by the existing Calico eBPF e2e test suite (
tests/e2e/calico_ebpf/calico_ebpf_test.go).Linked Issues
N/A
User-Facing Change
Further Comments
The Vagrantfile already computes each node's IP as
#{NETWORK4_PREFIX}.#{100+node_num}, so server-0 gets10.10.10.100. The same argument-passing pattern is already used by theconfigure_second_interface.shprovisioner in the same Vagrantfile.