Non-default domain name causing cilium hubble enable Hubble Relay to fail

[abcxcba]

------------------------------------------ Environment -----------------------------------------
kubernets Version: v1.30.1

kubeadm.conf:
networking:
dnsDomain: cluster.myk8s

install cilium:
cilium install --version 1.15.5

cilium status
/¯¯
/¯¯_/¯¯\ Cilium: OK
_/¯¯_/ Operator: OK
/¯¯_/¯¯\ Envoy DaemonSet: disabled (using embedded mode)
_/¯¯_/ Hubble Relay: disabled
__/ ClusterMesh: disabled

DaemonSet cilium Desired: 5, Ready: 5/5, Available: 5/5
Deployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1
Containers: cilium Running: 5
cilium-operator Running: 1
Cluster Pods: 2/2 managed by Cilium

cilium config view
...
hubble-disable-tls false
hubble-export-file-max-backups 5
hubble-export-file-max-size-mb 10
hubble-listen-address :4244
hubble-socket-path /var/run/cilium/hubble.sock
hubble-tls-cert-file /var/lib/cilium/tls/hubble/server.crt
hubble-tls-client-ca-files /var/lib/cilium/tls/hubble/client-ca.crt
hubble-tls-key-file /var/lib/cilium/tls/hubble/server.key
...

------------------------------------- Hubble ------------------------------
cilium hubble enable --relay --ui

kubectl get pod -A -o wide
kube-system hubble-relay-6446b75c8c-wh2vr 0/1 CrashLoopBackOff

log errors :
evel=info msg="Starting gRPC server..." options="{peerTarget:hubble-peer.kube-system.svc.cluster.local:443 dialTimeout:5000000000 ...

Direct cause of the error: ----> hubble-peer.kube-system.svc.cluster.local

==================Trying to solve problems=========================
cilium hubble disable
cilium config set hubble.peerService.clusterDomain "cluster.myk8s"
cilium config view | grep clusterDomain
hubble.peerService.clusterDomain cluster.myk8s

cilium hubble enable --relay --ui

The problem remains.

==================Solve problems=========================

cilium hubble disable

helm upgrade cilium cilium/cilium --version 1.15.5
--namespace kube-system
--reuse-values
--set hubble.peerService.clusterDomain="cluster.myk8s"
--set hubble.relay.enabled=true
--set hubble.ui.enabled=true

cilium hubble enable --relay --ui

kubectl get pod -A -o wide
kube-system hubble-relay-6446b75c8c-hcg2d 1/1 Running 0 32s

Hubble Relay pod is running successfully.

I don't know what the root cause of the problem is.

[dpsi4]

Hi There,

The simplest fix I had for this was:

# This first command leaves a broken Peering Service  
cilium hubble enable --relay --ui

# The following will fix the clusterDomain, and restart
cilium upgrade --reuse-values --helm-set hubble.peerService.clusterDomain=cluster1.local

Then the Hubble Peering service gets the correct clusterDomain and starts for me normally. I've been hitting this a while, so I have a README.md with this information as I try to tweak cilium to my needs.

I think the key difference you had was using set rather than helm-set. A good rule of thumb is that camelcase dot-divided keys are helm-set, while the lowercase hyphenated ones are set.