Skip to content

Reconcile HetznerBareMetalHosts when their robot credentials or SSH secrets change #2157

Description

@abdullah599

The Problem

CAPH reconciles each HetznerBareMetalHost by reading its credentials fresh from Kubernetes secrets on every pass. Those credentials come in two kinds, and both live in secrets that belong to the cluster or the machine, not to the host.

  • The Robot API user and password, in the Hetzner secret the cluster points to (HetznerCluster.spec.hetznerSecretRef), which also holds the hcloud token.
  • The SSH keys, a cluster-wide rescue key (HetznerCluster.spec.sshKeys.robotRescueSecretRef) and a per-machine key for the installed OS (HetznerBareMetalMachine.spec.sshSpec.secretRef, copied onto the host).

When a secret holds a wrong value, the two credential kinds fail differently:

  1. Wrong robot credentials stop the host. The Robot API answers unauthorized, so the controller marks RobotCredentialsAvailable false and returns without asking for a retry. Missing or empty credentials end the same way. The failure is not permanent, so any later reconcile would recover the host. But once the credentials are corrected, nothing triggers that reconcile, and the host stays stuck.
  2. A wrong SSH secret only slows the host. A missing one makes the host requeue every five minutes. A rotated key is then picked up on the next pass, late rather than never.

Note

The host controller's error path already assumes this watch exists. It returns with a comment that the host "will be triggered as soon as the secret is updated", but no such watch is registered today.

The Solution

Give the host controller a watch on secrets, so a corrected secret takes effect at once instead of leaving the host stuck or picking the fix up late. PR #2001 did this for HCloudMachines and the hcloud token in issue #1972.

  • Watch secrets in the host controller.
  • Reuse the predicate PR ✨ Reconcile HCloudMachines when hcloud token secret changes #2001 added, IgnoreInsignificantSecretUpdates, which ignores metadata-only updates and fires only when a secret's data changes.
  • When a secret changes, reconcile every host that reads it. That is every host of a HetznerCluster whose hetznerSecretRef or robotRescueSecretRef names the secret, plus any host whose own OS SSH secretRef names it.

The host controller is the only place that reads these credentials and calls the Robot API, so it is the only controller that needs the watch.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions