[Draft]: CSPL-3432:Add cert-manager Integration to Splunk Operator for Automatic TLS Certificate Management

[vivekr-splunk]

This Pull Request introduces cert-manager integration into the Splunk Operator, enabling automated TLS certificate provisioning and renewal for Splunk Pods. The changes allow users to specify whether they want to use:

1. CSI Driver – Automatically mounts cert-manager–managed certificates via the csi.cert-manager.io driver.
2. Sidecar Injection – Relies on a sidecar container (either added directly or via a sidecar injector webhook) to handle certificates and trigger Splunk to reload on rotation.

By setting certain annotations on the Splunk Custom Resource (CR), operators can choose the appropriate method for their environment. The Splunk Operator then modifies the final Pod spec accordingly—injecting the cert-manager CSI volume or sidecar references.

---

Key Changes

1. Annotation-Based Approach

   • Introduced a new annotation key splunk.com/cert-manager that can be set to either:
     • "csi" – Use the cert-manager CSI driver to mount certificates.
     • "injector" (or "sidecar") – Use a sidecar container approach (or an external sidecar injector webhook).

2. New Helper Functions

   • addCertManagerCsiVolume(...):

     • Adds a volume referencing the csi.cert-manager.io driver.
     • Ensures readOnly: true is set (required by cert-manager CSI).
     • Accepts annotation values like splunk.com/cert-secret-name, splunk.com/cert-issuer-name, etc. to populate volumeAttributes (e.g. csi.cert-manager.io/secret-name, csi.cert-manager.io/issuer-name, etc.).
     • Optionally sets a VolumeMount for /mnt/splunk/certificates (or any configured path) within the Splunk container.

   • addCertManagerSidecarInjector(...) (or addCertManagerSidecar(...) if manual injection):

     • In the manual injection scenario, it appends a container that watches the certificate directory (e.g. using inotifywait) and triggers Splunk to reload on changes.
     • In the webhook/injector scenario, it adds the required Pod annotations that a mutating admission webhook uses to inject the sidecar at runtime.

3. Pod Template Updates

   • Within updateSplunkPodTemplateWithConfig(...), the Operator checks if splunk.com/cert-manager is set.
   • Depending on the value, it calls the appropriate helper to configure the Pod’s volumes, annotations, and container specs.

4. Support for Key Reuse

   • If reuse-private-key: "true" is detected, the Operator expects secret-name to be specified so the private key can be stored in a persistent Secret.

5. Better Error Handling and Debug

   • Includes improved logging and checks to ensure required fields (e.g., issuerName, secretName) appear in the final Pod spec.
   • Addresses common CSI driver errors (read-only volume, missing issuerName, missing secretName, etc.).

---

How to Use

1. Set Up cert-manager: Make sure cert-manager and its CSI driver are installed.
2. Create or Reference an Issuer/ClusterIssuer: e.g., selfsigned-issuer or letsencrypt-prod.
3. Annotate Splunk CR:

   apiVersion: enterprise.splunk.com/v4
   kind: Standalone
   metadata:
     name: example-standalone-csi
     namespace: test
     annotations:
       splunk.com/cert-manager: "csi"
       splunk.com/cert-secret-name: "splunk-cert-secret"
       splunk.com/cert-issuer-name: "local-ca-issuer"
       splunk.com/cert-issuer-kind: "Issuer"
   spec:
     replicas: 1

   • The Operator now sets up a CSI volume or sidecar container accordingly, mounting valid TLS certs and reloading Splunk as needed.

---

Testing

• Unit Tests: Added coverage to ensure volume attributes for issuerName, secretName, and readOnly: true are correctly included in the generated Pod spec.
• Integration Testing: Verified that a Standalone Splunk CR with splunk.com/cert-manager: "csi" successfully mounts a certificate from csi.cert-manager.io and that reloading logic handles certificate updates.
• Regression Checks: Confirmed no disruptions to existing secret-based or self-signed Splunk deployments if the annotation is not set.

---

Additional Notes

• If sidecar injection is chosen, admins can use an external injector webhook or manually append a container with logic to detect changes in /mnt/splunk/certificates (or their chosen mount path).
• We’ve documented typical error messages (e.g., “issuer-name is a required field”, “secret name missing”) to help users quickly diagnose misconfiguration.

---

[coveralls]

Pull Request Test Coverage Report for Build 13671079310

Details

• 8 of 138 (5.8%) changed or added relevant lines in 2 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-0.9%) to 85.688%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/splunk/enterprise/configuration.go	8	15	53.33%
pkg/splunk/client/certmanager.go	0	123	0.0%

Totals
Change from base Build 13662127421:	-0.9%
Covered Lines:	10549
Relevant Lines:	12311

---

💛 - Coveralls

[coveralls]

Pull Request Test Coverage Report for Build 13671079317

Details

• 8 of 138 (5.8%) changed or added relevant lines in 2 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-0.9%) to 85.688%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/splunk/enterprise/configuration.go	8	15	53.33%
pkg/splunk/client/certmanager.go	0	123	0.0%

Totals
Change from base Build 13662127421:	-0.9%
Covered Lines:	10549
Relevant Lines:	12311

---

💛 - Coveralls

[rmouritzen-splunk]

This is installing cert-manager from the legacy location. From what I'm seeing online, the Jetstack version was decommissioned back in May 2025. Looks like this should use the newer location for the current open source fork of the project. See https://cert-manager.io/docs/installation/helm/

(I'm working on the Splunk Security Operator, which is why I came across this.)

Edit: This PR is from March 2025, so this looks like a minor case of bit-rot. Apparently CyberArk, the current maintainer, (indirectly) acquired JetStack, the original creator. Should be easy to fix.