Self Hosted: fixes

charts/controlplane/SELFHOSTED_INTRA_CLUSTER_AWS.md

@@ -6,7 +6,7 @@ This guide covers deploying Union control plane in the **same Kubernetes cluster
 
 In an intra-cluster deployment, the control plane and dataplane communicate using Kubernetes internal networking rather than external endpoints. This architecture simplifies networking, reduces costs, and provides complete data sovereignty.
 
-**Important**: This guide assumes you will also deploy the dataplane in the same cluster. See the [Dataplane Intra-Cluster Guide](../dataplane/SELFHOST_INTRA_CLUSTER_AWS.md) for dataplane-specific configuration.
+**Important**: This guide assumes you will also deploy the dataplane in the same cluster. See the [Dataplane Intra-Cluster Guide](../dataplane/SELFHOSTED_INTRA_CLUSTER_AWS.md) for dataplane-specific configuration.
 
 ## Benefits of Intra-Cluster Deployment
 
@@ -43,8 +43,9 @@ Choose standard hosted deployment when:
 
 2. **PostgreSQL database**:
    - Version: PostgreSQL 12+
-   - Can be AWS RDS or self-hosted in the cluster
-   - Required for all control plane services
+   - Can be AWS RDS or self-hosted in the cluster (it's not deployed by the Helm chart)
+   - Required for all control plane services. 
+   - Memory-optimized instances are recommended.
 
 3. **ScyllaDB** (for queue service):
    - Can be deployed via the Helm chart (embedded) or externally managed
@@ -58,6 +59,12 @@ Choose standard hosted deployment when:
    - Control plane services (with S3 access)
    - Artifacts service (with S3 access)
 
+6. **cert-manager**
+   - Used by the database to generate TLS certificate
+   - It can be added as Add-on to your cluster or installed by different methos, as covered in [cert-manager docs](https://cert-manager.io/docs/installation/)
+
+Check out the [deployment page](https://www.union.ai/docs/v1/selfmanaged/deployment/cluster-recommendations/#iam) for an example IAM policy.
+
 ### Required Tools
 
 - `kubectl` configured to access your cluster
@@ -76,8 +83,7 @@ Choose standard hosted deployment when:
 #### Install ScyllaDB CRDs (if using embedded ScyllaDB)
 
 ```bash
-cd helm-charts/charts/controlplane
-./scripts/install-scylla-crds.sh
+curl -O https://raw.githubusercontent.com/unionai/helm-charts/refs/heads/main/charts/controlplane/scripts/install-scylla-crds.sh && bash install-scylla-crds.sh
 ```
 
 #### Add Helm Repositories
@@ -111,9 +117,18 @@ kubectl create secret tls controlplane-tls-cert \
 
 **Option B: Using cert-manager (recommended for production)**
 
-See the example in `values.aws.selfhosted-intracluster.yaml` under the `extraObjects` section.
+See the example #3 in `values.aws.selfhosted-intracluster.yaml` under the `extraObjects` section.
+
+### Step 3: Create Database Password Secret
 
-### Step 3: Configure Values File
+```bash
+# Create secret with database password
+kubectl create secret generic union-controlplane-secrets \
+  --from-literal=pass.txt='YOUR_DB_PASSWORD' \
+  -n union-cp
+```
+
+### Step 4: Configure Values File
 
 Download and configure the intra-cluster values file:
 
@@ -124,14 +139,6 @@ curl -O https://raw.githubusercontent.com/unionai/helm-charts/main/charts/contro
 
 Edit `values.aws.selfhosted-intracluster.yaml` by setting all `global` values and replace all empty `""` values. This file is self-contained and includes all necessary AWS and intra-cluster configuration.
 
-### Step 4: Create Database Password Secret
-
-```bash
-# Create secret with database password
-kubectl create secret generic union-controlplane-secrets \
-  --from-literal=pass.txt='YOUR_DB_PASSWORD' \
-  -n union-cp
-```
 
 ### Step 5: Install Control Plane
 

charts/controlplane/scripts/install-scylla-crds.sh

@@ -78,6 +78,5 @@ kubectl apply --server-side -f "${CRD_DIR}/"
 echo ""
 echo "==> ✓ ScyllaDB Operator CRDs installed successfully!"
 echo ""
-echo "You can now install the control plane chart with scylla.enabled=true:"
-echo "  helm install my-controlplane ./controlplane -f values-scylla-example.yaml"
+echo "You can now proceed with the rest of the controlplane installation."
 echo ""

charts/controlplane/values.aws.selfhosted-intracluster.yaml

@@ -56,6 +56,7 @@ global:
 
   # Name of Kubernetes secret containing the DB password and other service specific secrets.
   # The secret can be created and set through databaseSecret.secretManifest and dbPass below.
+  # Check out example https://github.com/unionai/helm-charts/blob/main/charts/controlplane/values.yaml#L927
   # Example: "union-controlplane-secrets"
   # Note: Secret must contain "pass.txt" key
   KUBERNETES_SECRET_NAME: ""

charts/controlplane/values.yaml

@@ -921,8 +921,9 @@ flyte:
       # flyte-org/flyte flyte-core helm chart _helpers.tpl does not render templates.
       # Therefore we have to explicitly set the value here.
       # Ref: https://github.com/flyteorg/flyte/pull/6711
-      # TODO (DIRECTLY CONFIGURE): Match value to global.KUBERNETES_SECRET_NAME
-      name: "<KUBERNETES_SECRET_NAME>"
+      # IMPORTANT: Override this value directly in your values file (e.g., values.aws.yaml)
+      # Set to your actual secret name, NOT a template variable
+      name: ""
       # -- Leave it empty if your secret already exists
       secretManifest: {}
 

charts/dataplane/values.aws.selfhosted-intracluster.yaml

@@ -346,7 +346,6 @@ fluentbit:
     name: fluentbit-system
     # Annotations may be required for IRSA
     annotations: {}
-<<<<<<< Updated upstream
 
 # ----------------------------------------------------------------------------
 # SECTION 7: Task Level Monitoring

tests/generated/controlplane.aws.billing-enable.yaml

@@ -2214,8 +2214,7 @@ spec:
             capabilities:
               drop: ["ALL"]
           volumeMounts:
-          - mountPath: /etc/db
-            name: <KUBERNETES_SECRET_NAME>
+
           - mountPath: /etc/flyte/config
             name: base-config-volume
         - command:
@@ -2234,8 +2233,7 @@ spec:
             capabilities:
               drop: ["ALL"]
           volumeMounts:
-          - mountPath: /etc/db
-            name: <KUBERNETES_SECRET_NAME>
+
           - mountPath: /etc/flyte/config
             name: base-config-volume
         - name: generate-secrets
@@ -2305,8 +2303,7 @@ spec:
           capabilities:
             drop: ["ALL"]
         volumeMounts:
-        - mountPath: /etc/db
-          name: <KUBERNETES_SECRET_NAME>
+
         - mountPath: /srv/flyte
           name: shared-data
         - mountPath: /etc/flyte/config
@@ -2315,9 +2312,7 @@ spec:
           name: admin-secrets
       serviceAccountName: flyteadmin
       volumes:
-      - name: <KUBERNETES_SECRET_NAME>
-        secret:
-          secretName: <KUBERNETES_SECRET_NAME>
+
       - emptyDir: {}
         name: shared-data
       - emptyDir: {}
@@ -2582,8 +2577,7 @@ spec:
         imagePullPolicy: "IfNotPresent"
         name: run-migrations
         volumeMounts:
-        - mountPath: /etc/db
-          name: <KUBERNETES_SECRET_NAME>
+
         - mountPath: /etc/cacheservice/config
           name: config-volume
         securityContext:
@@ -2616,15 +2610,12 @@ spec:
             ephemeral-storage: 200Mi
             memory: 200Mi
         volumeMounts:
-        - mountPath: /etc/db
-          name: <KUBERNETES_SECRET_NAME>
+
         - mountPath: /etc/cacheservice/config
           name: config-volume
       serviceAccountName: cacheservice
       volumes:
-      - name: <KUBERNETES_SECRET_NAME>
-        secret:
-          secretName: <KUBERNETES_SECRET_NAME>
+
       - emptyDir: {}
         name: shared-data
       - configMap:

tests/generated/controlplane.aws.yaml

@@ -2214,8 +2214,7 @@ spec:
             capabilities:
               drop: ["ALL"]
           volumeMounts:
-          - mountPath: /etc/db
-            name: <KUBERNETES_SECRET_NAME>
+
           - mountPath: /etc/flyte/config
             name: base-config-volume
         - command:
@@ -2234,8 +2233,7 @@ spec:
             capabilities:
               drop: ["ALL"]
           volumeMounts:
-          - mountPath: /etc/db
-            name: <KUBERNETES_SECRET_NAME>
+
           - mountPath: /etc/flyte/config
             name: base-config-volume
         - name: generate-secrets
@@ -2305,8 +2303,7 @@ spec:
           capabilities:
             drop: ["ALL"]
         volumeMounts:
-        - mountPath: /etc/db
-          name: <KUBERNETES_SECRET_NAME>
+
         - mountPath: /srv/flyte
           name: shared-data
         - mountPath: /etc/flyte/config
@@ -2315,9 +2312,7 @@ spec:
           name: admin-secrets
       serviceAccountName: flyteadmin
       volumes:
-      - name: <KUBERNETES_SECRET_NAME>
-        secret:
-          secretName: <KUBERNETES_SECRET_NAME>
+
       - emptyDir: {}
         name: shared-data
       - emptyDir: {}
@@ -2582,8 +2577,7 @@ spec:
         imagePullPolicy: "IfNotPresent"
         name: run-migrations
         volumeMounts:
-        - mountPath: /etc/db
-          name: <KUBERNETES_SECRET_NAME>
+
         - mountPath: /etc/cacheservice/config
           name: config-volume
         securityContext:
@@ -2616,15 +2610,12 @@ spec:
             ephemeral-storage: 200Mi
             memory: 200Mi
         volumeMounts:
-        - mountPath: /etc/db
-          name: <KUBERNETES_SECRET_NAME>
+
         - mountPath: /etc/cacheservice/config
           name: config-volume
       serviceAccountName: cacheservice
       volumes:
-      - name: <KUBERNETES_SECRET_NAME>
-        secret:
-          secretName: <KUBERNETES_SECRET_NAME>
+
       - emptyDir: {}
         name: shared-data
       - configMap: