[e2e test flake] Timed waiting for all control plane replicas to be updated

[cahillsf]

Which jobs are flaking?

periodic-cluster-api-e2e-mink8s-main
periodic-cluster-api-e2e-main

Which tests are flaking?

When testing KCP adoption Should adopt up-to-date control plane Machines without modification

Since when has it been flaking?

2024-11-05 is the first instance of this flake i can find: https://storage.googleapis.com/k8s-triage/index.html?date=2024-11-06&text=Timed%20waiting%20for%20all%20control%20plane%20replicas%20to%20be%20updated&job=.*periodic-cluster-api-e2e.*

now it appears quite consistently:
https://storage.googleapis.com/k8s-triage/index.html?date=2024-11-20&text=Timed%20waiting%20for%20all%20control%20plane%20replicas%20to%20be%20updated&job=.*periodic-cluster-api-e2e.*

Testgrid link

https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api#capi-e2e-main

Reason for failure (if possible)

TBD (have not had time to investigate thoroghly)

Anything else we need to know?

example failed job: https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/periodic-cluster-api-e2e-mink8s-main/1858697443558821888

have filtered commits on main to around the time this flake popped up: https://github.com/kubernetes-sigs/cluster-api/commits/main/?since=2024-11-03&until=2024-11-05

have not have time to further investigate yet

Label(s) to be applied

/kind flake

[cahillsf]

/area e2e-testing

[sivchari]

I confirm this flake.
/assign

[adilGhaffarDev]

did we fix something related to this? I don't see any failure after 23-11
https://storage.googleapis.com/k8s-triage/index.html?text=Timed%20waiting%20for%20all%20control%20plane%20replicas%20to%20be%20updated&job=.*-cluster-api-.*&xjob=.*-provider-.*%7C.*-operator-.*

[chrischdi]

New occurencies now.

Seems not to be gone.

[cahillsf]

@sivchari 👋 are you still working on this?

[sivchari]

Hi @cahillsf , I didn't have time to work on this. I'll try it since next week. Or should I unassign myself ? Thanks.

[cahillsf]

Hi @cahillsf , I didn't have time to work on this. I'll try it since next week. Or should I unassign myself ? Thanks.

if you have time please go ahead, even if you dont have a fix would be helpful to add findings. if youre not going to have time then feel free to unassign and we can find some assignee from the CI release team for this cycle

[sivchari]

Hi, @cahillsf
Once, I unassign me.
/unassign