feat!: Add working conversion webhook with cert rotation #1066

sbernauer · 2025-06-30T14:07:23Z

Description

An working example usage can be found in stackabletech/zookeeper-operator#958 (mainly look at rust/operator-binary/src/main.rs)

Definition of Done Checklist

Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
Please make sure all these things are done and tick the boxes

Author

Changes are OpenShift compatible
CRD changes approved
CRD documentation for all fields, following the style guide.
Integration tests passed (for non trivial changes)
Changes need to be "offline" compatible

Reviewer

Code contains useful comments
Code contains useful logging statements
(Integration-)Test cases added
Documentation added or updated. Follows the style guide.
Changelog updated
Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

Feature Tracker has been updated
Proper release label has been added

sbernauer · 2025-07-02T06:11:49Z

crates/stackable-webhook/src/tls/mod.rs

+pub const WEBHOOK_CA_LIFETIME: Duration = Duration::from_minutes_unchecked(3);
+pub const WEBHOOK_CERTIFICATE_LIFETIME: Duration = Duration::from_minutes_unchecked(2);
+pub const WEBHOOK_CERTIFICATE_ROTATION_INTERVAL: Duration = Duration::from_minutes_unchecked(1);


reminder to bump these before merging. Currently they are so low for easy testing

Techassi

Partial review, I didn't look at the CertificateResolver yet.

crates/stackable-operator/src/cli.rs

crates/stackable-webhook/src/lib.rs

crates/stackable-webhook/src/servers/conversion.rs

Co-authored-by: Techassi <[email protected]>

crates/stackable-webhook/src/tls/mod.rs

Co-authored-by: Techassi <[email protected]>

Techassi

I left a bunch of comments.

I encountered the ConversionWebhookServer which is missing the changes we talked about a few weeks back. As such, it doesn't make a whole lot of sense to continue the review before these changes are implemented. I've sent you an appropriate patch as a private message which should be a good starting point for the changes we discussed.

Techassi · 2025-08-06T14:31:10Z

crates/stackable-operator/src/cli.rs

+            "--operator-namespace",
+            "stackable-operators",
+            "--operator-service-name",
+            "foo-operator",


note: I still feel like all these CLI unit tests are pretty much useless and should be removed to speed up the testing runs.

crates/stackable-operator/CHANGELOG.md

Techassi · 2025-08-06T14:46:07Z

crates/stackable-webhook/src/servers/conversion.rs

+use kube::{
+    Api, Client, ResourceExt,
+    api::{Patch, PatchParams},
+};
+use snafu::{OptionExt, ResultExt, Snafu};
+use stackable_operator::cli::OperatorEnvironmentOptions;
+use tokio::{sync::mpsc, try_join};
 use tracing::instrument;
+use x509_cert::{
+    Certificate,
+    der::{EncodePem, pem::LineEnding},
+};


note: Please combine these imports with the others above.

hmm, what imports exactly can be combined? rustfmt groups them like this

Yeah this is sub-optimal formatting sadly. You need to manually move them to the top imports above line 11.

crates/stackable-webhook/src/servers/conversion.rs

Techassi

We are mostly there, just some small things left.

Techassi · 2025-08-13T12:31:10Z

crates/stackable-operator/CHANGELOG.md

@@ -4,6 +4,17 @@ All notable changes to this project will be documented in this file.

 ## [Unreleased]

+### Added
+
+- BREAKING: Add two new required CLI arguments: `--operator-namespace` and `-operator-service-name`.


Suggested change

- BREAKING: Add two new required CLI arguments: `--operator-namespace` and `-operator-service-name`.

- BREAKING: Add two new required CLI arguments: `--operator-namespace` and `--operator-service-name`.

Techassi · 2025-08-13T12:44:10Z

crates/stackable-webhook/src/servers/conversion.rs

+    RunWebhookServer { source: WebhookError },
+
+    #[snafu(display("failed to receive certificate from channel"))]
+    ReceiverCertificateFromChannel,


Suggested change

ReceiverCertificateFromChannel,

ReceiveCertificateFromChannel,

Techassi · 2025-08-13T12:46:21Z

crates/stackable-webhook/src/servers/conversion.rs

+    /// The environment the operator is running in, notably the namespace and service name it is
+    /// reachable at.
+    pub operator_environment: OperatorEnvironmentOptions,


note: Let's avoid pulling in a type from stackable_operator here. Instead, split it into two separate fields.

Suggested change

/// The environment the operator is running in, notably the namespace and service name it is

/// reachable at.

pub operator_environment: OperatorEnvironmentOptions,

pub namespace: String,

pub service_name: String,

Techassi · 2025-08-13T12:54:08Z

crates/stackable-webhook/src/servers/mod.rs

@@ -2,4 +2,5 @@
 //! purposes.
 mod conversion;

-pub use conversion::*;
+pub use conversion::{ConversionWebhookError, ConversionWebhookOptions, ConversionWebhookServer};
+pub use kube::core::conversion::ConversionReview;


note: This is already re-exported via stackable_webhook::servers::conversion::ConversionReview. As such, please remove it from here again.

Techassi · 2025-08-13T13:18:19Z

crates/stackable-webhook/src/tls/cert_resolver.rs

+            let certificate = ca
+                .generate_ecdsa_leaf_certificate(
+                    "Leaf",
+                    "webhook",
+                    subject_alterative_dns_names.iter().map(|san| san.as_str()),
+                    WEBHOOK_CERTIFICATE_LIFETIME,
+                )
+                .context(GenerateLeafCertificateSnafu)?;
+
+            let certificate_der = certificate
+                .certificate_der()
+                .context(EncodeCertificateDerSnafu)?;
+            let private_key_der = certificate
+                .private_key_der()
+                .context(EncodePrivateKeyDerSnafu)?;
+            let certificate_key =
+                CertifiedKey::from_der(vec![certificate_der], private_key_der, &tls_provider)
+                    .context(DecodeCertifiedKeyFromDerSnafu)?;
+
+            Ok((certificate.certificate().clone(), Arc::new(certificate_key)))


suggestion: Rename this to certificate_pair to better indicate what this is.

Suggested change

let certificate = ca

.generate_ecdsa_leaf_certificate(

"Leaf",

"webhook",

subject_alterative_dns_names.iter().map(|san| san.as_str()),

WEBHOOK_CERTIFICATE_LIFETIME,

)

.context(GenerateLeafCertificateSnafu)?;

let certificate_der = certificate

.certificate_der()

.context(EncodeCertificateDerSnafu)?;

let private_key_der = certificate

.private_key_der()

.context(EncodePrivateKeyDerSnafu)?;

let certificate_key =

CertifiedKey::from_der(vec![certificate_der], private_key_der, &tls_provider)

.context(DecodeCertifiedKeyFromDerSnafu)?;

Ok((certificate.certificate().clone(), Arc::new(certificate_key)))

let certificate_pair = ca

.generate_ecdsa_leaf_certificate(

"Leaf",

"webhook",

subject_alterative_dns_names.iter().map(|san| san.as_str()),

WEBHOOK_CERTIFICATE_LIFETIME,

)

.context(GenerateLeafCertificateSnafu)?;

let certificate_der = certificate_pair

.certificate_der()

.context(EncodeCertificateDerSnafu)?;

let private_key_der = certificate_pair

.private_key_der()

.context(EncodePrivateKeyDerSnafu)?;

let certificate_key =

CertifiedKey::from_der(vec![certificate_der], private_key_der, &tls_provider)

.context(DecodeCertifiedKeyFromDerSnafu)?;

Ok((certificate_pair.certificate().clone(), Arc::new(certificate_key)))

Techassi · 2025-08-13T13:37:11Z

crates/stackable-webhook/src/tls/cert_resolver.rs

+        let (cert, certified_key) = Self::generate_new_cert(subject_alterative_dns_names.clone())
+            .await
+            .context(GenerateNewCertificateSnafu)?;
+
+        cert_tx
+            .send(cert)
+            .await
+            .map_err(|_err| CertificateResolverError::SendCertificateToChannel)?;


note: I feel like this can be moved into its own function, because we repeat the exact same code below.

Techassi · 2025-08-13T13:38:43Z

crates/stackable-webhook/src/options.rs

 ///
 /// Usually, this struct is not constructed manually, but instead by calling
-/// [`Options::builder()`] or [`OptionsBuilder::default()`].
+/// [`WebhookOptions::builder()`] or [`OptionsBuilder::default()`].
 #[derive(Debug, Default)]
 pub struct OptionsBuilder {


note: We renamed Options to WebhookOptions. As such, I think we should also rename the builder accordingly.

Suggested change

pub struct OptionsBuilder {

pub struct WebhookOptionsBuilder {

Techassi · 2025-08-13T13:42:32Z

crates/stackable-webhook/CHANGELOG.md

+### Added
+
+- BREAKING: Re-write the `ConversionWebhookServer`.
+  It can now do CRD conversions, handle multiple CRDs and takes care of reconciling the CRDs ([#1066]).


note: I still feel like the webhhok should not take care of reconciling the CRDs, but the operator should do it instead. Let's tackle this in a follow-up PR and leave it as is to get something out of the door.

Techassi · 2025-08-13T13:43:56Z

crates/stackable-webhook/CHANGELOG.md

@@ -4,6 +4,23 @@ All notable changes to this project will be documented in this file.

 ## [Unreleased]

+### Added


note: I feel like this should go under the "Changed" section instead. We additionally need to mention the Options renames.

Techassi · 2025-08-13T13:45:03Z

crates/stackable-webhook/CHANGELOG.md

+  Also, `TlsServer::new` now returns an additional `mpsc::Receiver<Certificate>`, so that the caller
+  can get notified about certificate rotations happening ([#1066]).


note: (Addition to the comment above) This would be the perfect trigger for the operator (the caller if this function) to reconcile the CRDs.

sbernauer added 9 commits June 27, 2025 17:11

WIP

0ef29ab

Actually rotate certs

ed58563

clippy

3be083e

doctests

fd4d94b

HashMap -> Vec

b3c17b3

Set correct CA lifetime

bdbcee0

Result type

b4c22b0

fix rustdocs

31b128b

Add some docs

b84cf05

sbernauer mentioned this pull request Jun 30, 2025

tracking: CRD versioning stackabletech/issues#642

Open

88 tasks

Update rustdoc

0390680

sbernauer changed the title ~~feat(webhook): Add functioning conversion webhook with cert rotation~~ feat(webhook): Add working conversion webhook with cert rotation Jul 2, 2025

sbernauer added 3 commits July 2, 2025 08:09

Handle cert rottion loop falures

2e1d085

changelog

3f8e744

link to decision

6f0040c

sbernauer commented Jul 2, 2025

View reviewed changes

sbernauer mentioned this pull request Jul 2, 2025

feat!: Add conversion webhook stackabletech/zookeeper-operator#958

Draft

20 tasks

sbernauer self-assigned this Jul 2, 2025

fix rustdocs

941ee6b

sbernauer added this to Stackable Engineering Jul 2, 2025

sbernauer moved this to Development: In Progress in Stackable Engineering Jul 2, 2025

sbernauer moved this from Development: In Progress to Development: In Review in Stackable Engineering Jul 2, 2025

sbernauer moved this from Development: In Review to Development: Waiting for Review in Stackable Engineering Jul 2, 2025

sbernauer mentioned this pull request Jul 2, 2025

feat: Add Service on top of operators stackabletech/operator-templating#534

Open

Techassi self-requested a review July 3, 2025 06:43

Techassi moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Jul 3, 2025

sbernauer mentioned this pull request Jul 3, 2025

Use the local kubelet's cluster domain, rather than a random one #1071

Merged

8 tasks

Techassi requested changes Jul 3, 2025

View reviewed changes

Techassi changed the title ~~feat(webhook): Add working conversion webhook with cert rotation~~ feat!: Add working conversion webhook with cert rotation Jul 3, 2025

Merge remote-tracking branch 'origin/main' into feat/conversion-webhook

0c52f5f

sbernauer and others added 9 commits July 4, 2025 10:12

Remove mpsc in tests leftover

a3b3cc1

docs: Use result of WebhookServer::new

f8683e4

fmt

457dbf3

Update crates/stackable-webhook/src/tls/mod.rs

a39ee47

Co-authored-by: Techassi <[email protected]>

fix suggestion

6210262

capture in variable

dcb22bd

Move route into variable

f1fee5d

docs: Mention background cert rotation

caddffb

docs: Document TlsServer::new

7450d85

sbernauer requested a review from Techassi July 4, 2025 10:19

sbernauer and others added 5 commits July 4, 2025 12:23

docs: Hint on downward API env var

ac29c0e

Rewrite certificate rotation logic

51321f0

Co-authored-by: Techassi <[email protected]>

Avoid unwrap

082a59d

Make CertResolver pub, so docs pass

c032633

Add missing await

8dedf5a

Techassi reviewed Jul 8, 2025

View reviewed changes

crates/stackable-webhook/src/tls/mod.rs Outdated Show resolved Hide resolved

sbernauer and others added 3 commits July 8, 2025 15:56

mount -> project

9da5588

Update crates/stackable-webhook/src/tls/mod.rs

758202f

Co-authored-by: Techassi <[email protected]>

changelog

29b0c6c

This was referenced Jul 9, 2025

feat!(stackable-certs): Use builder pattern and support SAN entries #1044

Closed

feat!: Re-export stackable-certs and stackable-webhook crates #1074

Open

Techassi requested changes Aug 6, 2025

View reviewed changes

sbernauer added 5 commits August 11, 2025 17:10

Add ConversionWebhookOptions struct

8b83131

Merge remote-tracking branch 'origin/main' into feat/conversion-webhook

e9e182e

PascalCase error variants

fe1749d

changelog

3894d3b

changelog

0936315

sbernauer requested a review from Techassi August 11, 2025 15:36

fix rustdocs

99f1dfe

Techassi requested changes Aug 13, 2025

View reviewed changes

	- BREAKING: Add two new required CLI arguments: `--operator-namespace` and `-operator-service-name`.
	- BREAKING: Add two new required CLI arguments: `--operator-namespace` and `--operator-service-name`.

	ReceiverCertificateFromChannel,
	ReceiveCertificateFromChannel,

	pub struct OptionsBuilder {
	pub struct WebhookOptionsBuilder {

		@@ -4,6 +4,23 @@ All notable changes to this project will be documented in this file.

		## [Unreleased]

		### Added

		Also, `TlsServer::new` now returns an additional `mpsc::Receiver<Certificate>`, so that the caller
		can get notified about certificate rotations happening ([#1066]).

Uh oh!

feat!: Add working conversion webhook with cert rotation #1066

Are you sure you want to change the base?

feat!: Add working conversion webhook with cert rotation #1066

Conversation

sbernauer commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Definition of Done Checklist

Author

Reviewer

Acceptance

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Techassi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Techassi left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Techassi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sbernauer commented Jun 30, 2025 •

edited

Loading

Techassi left a comment •

edited

Loading