Skip to content

e2e: use server node IP for Calico eBPF kubernetes endpoint #10936

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
thomasferrandiz wants to merge 1 commit into
base: master
Choose a base branch
from
Open

e2e: use server node IP for Calico eBPF kubernetes endpoint #10936

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion tests/e2e/calico_ebpf/Vagrantfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ def provision(vm, roles, role_num, node_num)

scripts_location = Dir.exist?("./scripts") ? "./scripts" : "../scripts"
vm.provision "Configure second interface", type: "shell", path: scripts_location + "/configure_second_interface.sh", args: [ node_ip4, node_ip6, node_ip6_gw, vm.box ]
vm.provision "Create Calico Manifest", type: "shell", path: scripts_location + "/calico_ebpf_manifest.sh"
vm.provision "Create Calico Manifest", type: "shell", path: scripts_location + "/calico_ebpf_manifest.sh", args: [ node_ip4 ]

install_type = getInstallType(vm, RELEASE_VERSION, GITHUB_BRANCH)

Expand Down
4 changes: 3 additions & 1 deletion tests/e2e/scripts/calico_ebpf_manifest.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
# Set Calico parameters to use the eBPF dataplane instead of iptables
mkdir -p /var/lib/rancher/rke2/server/manifests

NODE_IP=${1}

echo "Creating calico chart"
echo "apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
Expand All @@ -20,4 +22,4 @@ spec:
kubeProxyManagement: Enabled
linuxDataplane: BPF
kubernetesServiceEndpoint:
host: localhost" > /var/lib/rancher/rke2/server/manifests/rke2-calico-config.yaml
host: ${NODE_IP}" > /var/lib/rancher/rke2/server/manifests/rke2-calico-config.yaml

@brandond brandond Jul 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really tell people to hardcode the IP of a single node in the chart values? That is terrible, we must not do that. Won't this break the calico for the whole cluster if that one node is down?

Why can't we use localhost:6443? This is guaranteed to be available on both servers and agents - if the node is not a control-plane node, there's a supervisor load-balancer listening on that port that forwards to an apiserver. This is build in to RKE2 and does not rely on kube-proxy.

@thomasferrandiz thomasferrandiz Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've had reports from customers (SURE-11652 for example) that using localhost doesn't work in that case.
The Calico pod calico-kube-controllers is not using the host network so it doesn't resolve localhost:6643 as the load-balancer endpoint.

That's also why I added the doc about using HAProxy for when users need HA with Calico eBPF.

@brandond brandond Jul 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. These are all ugly solutions that we should come up with a better answer for.

Can we modify calico-kube-controllers so that it does run with host network? Or does it need to be in the cluster for some reason?

If running with host network is not desirable for some reason, we should use downward API to inject the IP of the node the pod is running on (status.hostIP), instead of hardcoding a single IP in the values.

@thomasferrandiz thomasferrandiz Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't easily modify calico-kube-controllers because it's deployed by the Tigera Operator so we don't control the daemonset configuration.
I'll check if I can find a way to use status.hostIP in the KubernetesEndPoint configmap.

@thomasferrandiz thomasferrandiz Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like we can't do that either without changing the operator itself.
We could always open an issue upstream but there was projectcalico/calico#9141 already.

Their opinion seems to be that it's OK to have a non-redundant connection to the API server because the downtime would be short anyway.

Maybe kube-vip would be a better fit for this than Haproxy?

@brandond brandond Jul 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a really poor design in general... either you need to make one of your nodes a single point of failure for the whole cluster, or you need to stand up your own LB in front of the apiserver?

If it would just look at the Kubernetes Endpoints/EndpointSlice instead of the service ClusterIP, it would get the IPs of all the control-plane nodes, and could connect to them directly without having to do anything at all.

@thomasferrandiz thomasferrandiz Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree and we could try to submit a PR upstream but it doesn't look like they care.
eBPF is most likely there to check a box compared to Cilium it's not their main data plane.

Loading