fix(kubestellar): stop KubeFlex ControlPlane status hot loop

argocd/kubestellar-app.yaml

@@ -12,7 +12,7 @@ spec:
   source:
     repoURL: ghcr.io/kubestellar/kubestellar
     chart: core-chart
-    targetRevision: 0.29.0
+    targetRevision: 0.30.0
     helm:
       valuesObject:
         # its1 type host reuses ghost's API server (OCM hub in-cluster).
@@ -31,8 +31,14 @@ spec:
         kubeflex-operator:
           installPostgreSQL: false
         # External reachability deferred per ADR-0004: in-cluster Service
-        # DNS suffices until external WECs exist. KubeFlex-generated nginx
-        # Ingress objects are inert (no controller claims them).
+        # DNS suffices until external WECs exist. KubeFlex v0.9.3 still
+        # generates an nginx Ingress for each hosted control plane even though
+        # this app has no ingress controller. ingress-nginx reached EOL in
+        # March 2026; do not install or enable it to satisfy this artifact.
+        #
+        # 0.30.0 carries KubeFlex v0.9.3. Verify the controller's status-update
+        # behavior after rollout; if conflicts persist, escalate the upstream
+        # KubeFlex controller rather than installing an unrelated ingress.
   destination:
     server: https://kubernetes.default.svc
     namespace: kubestellar

docs/ops/RUNBOOK.md

@@ -169,6 +169,7 @@ Argo Workflow (argo namespace)
 | Service-catalog namespace stuck terminating | Finalizer or PVC not released | Check for stuck PVCs or pods with `kubectl get all -n <ns>`, delete manually if needed |
 | `lab-infra` sync wedged "waiting for healthy state of DaemonSet/..." | A DaemonSet pod is unhealthy on some node (e.g. hostPath missing on that host), blocking every subsequent manifests/ change | Fix or scope the DaemonSet (capability-label nodeSelector), then terminate the stuck operation so ArgoCD retries: `kubectl patch application lab-infra -n argocd --type=merge -p '{"status":{"operationState":{"phase":"Terminating"}}}'` |
 | KubeStellar app sync stuck at kubeflex-controller-manager | Postgres hook deadlock under ArgoCD | Keep `installPostgreSQL: false` + separate `kubestellar-postgres` app; see `docs/skills/kubestellar/SKILL.md` |
+| `kubeflex-controller-manager` consumes sustained host RX and repeats ControlPlane reconciliation about once per second | KubeFlex v0.9.1's status writes race across infrastructure, PostCreateHook, and final-readiness phases. The same controller unconditionally creates a `wds1` Ingress with `ingressClassName: nginx` even though the lab has no external WEC endpoint; the Ingress is outside PostCreateHook templates, has no address, and [upstream Kubernetes freezes the Ingress API in favor of Gateway](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/). The [Ingress NGINX retirement statement](https://kubernetes.io/blog/2026/01/29/ingress-nginx-statement/) says the controller receives no further fixes or security patches | Upgrade `argocd/kubestellar-app.yaml` to core-chart 0.30.0 (KubeFlex v0.9.3), reconcile through ArgoCD, and verify the controller logs plus Prometheus `container_network_receive_bytes_total` rate. Do not install ingress-nginx, add an IngressClass, or introduce a Gateway controller solely to claim this unused artifact. If external reachability is needed later, use the [Gateway API getting-started path](https://gateway-api.sigs.k8s.io/guides/getting-started/). If the status-conflict loop survives v0.9.3, escalate it as an upstream KubeFlex controller defect; deleting the generated Ingress alone will not persist |
 | `test-lane` nodes sit `Pending` for tens of minutes with no pods created, and `kubectl get wf <name> -o json \| jq .status.synchronization` shows `waiting` on `ghost-container-qa` | A single QA pipeline held most of the 6 semaphore slots. `spec.parallelism` is not inherited through `templateRef`, so poller-dispatched runs (pr-poller's inline `pr-pipeline`, `image-poller`) fanned out every `withItems` lane at once | The `pipeline` templates now carry template-level `parallelism: 2`, which survives `templateRef`. Never fix this by raising `ghost-container-qa`; that only moves the threshold. Verify with `python3 scripts/check_semaphore_topology.py argo/` and see [patterns §15b](../skills/argo-workflows/patterns.md) |
 | Several workflows running concurrently for the same PR at different SHAs, or workflows still running for merged PRs | The poller deduped on `pr-number` + `pr-sha` and never cancelled superseded or closed-PR runs; each stale run held a `ghost-container-qa` slot for ~20 minutes | `pr-poller` now supersedes (newest SHA wins) on every poll and reaps workflows whose PR left the open set, using `spec.shutdown: Stop` so `report-final` still publishes a terminal `ghost-lab` status. Inspect with `kubectl get wf -n argo -l bluefin.io/trigger=pr-auto -L bluefin.io/repository,bluefin.io/pr-number,bluefin.io/pr-sha`. Reaping is skipped for any repo whose open-PR enumeration hit an API error or returned zero PRs. See [patterns §20ac](../skills/argo-workflows/patterns.md) |
 | A PR's `ghost-lab` status is stuck on `pending` after its workflow disappeared | The workflow was hard-deleted, so its `onExit` `report-final` handler never ran | Never `kubectl delete` a live PR workflow; use `kubectl patch workflow <name> -n argo --type merge -p '{"spec":{"shutdown":"Stop"}}'`. To clear an already-stranded status, re-run the poller with `refresh-existing=true` |

docs/skills/kubestellar/SKILL.md

@@ -8,6 +8,7 @@ description: >
 metadata:
   context7-sources:
     - /kubestellar/kubestellar
+    - /argoproj/argo-cd
     - /websites/argo-cd_readthedocs_io_en_stable
     - /websites/prometheus_io
 ---
@@ -172,6 +173,8 @@ instance.
 | Prometheus cAdvisor targets stay `unknown` and the pod restarts | check for `OOMKilled`; WAL replay plus the controller and cAdvisor scrape set needs the committed 512 MiB request and 2 GiB limit, not the original demo sizing |
 | Prometheus reaches its memory limit after NFD labels appear | never `labelmap` all `__meta_kubernetes_node_label_*` values onto cAdvisor series; map only `__meta_kubernetes_node_name` to `node`, or NFD's 100+ labels multiply across every container metric |
 | Prometheus ConfigMap is Synced but scrape behavior does not change | bump `lab.projectbluefin.io/config-version` on the Deployment pod template with every scrape-config change; Prometheus has no config reloader sidecar |
+| `kubeflex-controller-manager` sustains a roughly 1 Hz ControlPlane loop, reports `failed to update final status ... object has been modified`, and looks like external bandwidth | KubeFlex v0.9.1 writes ControlPlane status during infrastructure, post-create-hook, and final readiness phases; the resulting status-update race requeues the controller. KubeFlex also generates an `ingressClassName: nginx` Ingress outside the PostCreateHooks, but does not inspect its load-balancer status; the ControlPlane can be `Ready=True` while the Ingress has an empty status. That endpoint is unused in this internal-only lab. Upstream Kubernetes says [Ingress is frozen and recommends Gateway](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/), and the [Ingress NGINX retirement statement](https://kubernetes.io/blog/2026/01/29/ingress-nginx-statement/) says there will be no post-retirement fixes or security patches | Upgrade the core chart to 0.30.0, which carries KubeFlex v0.9.3, and let ArgoCD roll the operator. Keep external reachability off by default; never install ingress-nginx or add a class solely to satisfy KubeFlex's hardcoded artifact. Verify `kubectl -n kubeflex-system logs deploy/kubeflex-controller-manager --tail=50` has no status-conflict loop and compare the `container_network_receive_bytes_total` rate before/after. If conflicts persist on v0.9.3, the remaining fix belongs upstream in KubeFlex rather than in an ingress manifest |
+| KubeFlex creates an `ingressClassName: nginx` Ingress in an internal-only k3s lab | The hosted ControlPlane reconcilers create this endpoint unconditionally when `isOpenShift=false`; current KubeFlex chart values expose no disable switch, and the object is not a PostCreateHook template. ADR-0004 intentionally provides no external endpoint. Upstream Kubernetes says [Ingress is frozen and recommends Gateway](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/), while the [Ingress NGINX retirement statement](https://kubernetes.io/blog/2026/01/29/ingress-nginx-statement/) says the retired controller receives no further fixes or security patches | Do not install ingress-nginx or a replacement Gateway controller just to claim the object. Keep the lab's off-by-default network policy and track an upstream KubeFlex option/removal; deleting the object alone only causes the owner controller to recreate it. If external reachability is needed later, follow the [Gateway API getting-started guidance](https://gateway-api.sigs.k8s.io/guides/getting-started/) rather than adding another Ingress |
 | `clusteradm get token` forbidden | workflow ran as `argo` SA; needs `serviceAccountName: kubestellar-bootstrap` |
 | Workflow pod rejected "failed quota: argo-quota" | missing resources requests/limits on the template |
 | Downsynced namespace exists but is empty | objectSelectors don't match the inner objects' labels |
@@ -204,3 +207,4 @@ every core upgrade.
 - [ ] `its1` and `wds1` report Ready
 - [ ] The target ManagedCluster reports Joined and Available
 - [ ] `kubestellar-smoke-test` passes after a core upgrade
+- [ ] KubeFlex logs remain free of the ControlPlane status-conflict loop after a core-chart upgrade