Dedupe env var keys between Helm ConfigMap and ExternalSecret #2420

Summary

Several env var keys are declared in both the Helm ConfigMap and the cluster ExternalSecret.
Every deployment lists secretRef after configMapRef in envFrom, and later sources win in
Kubernetes, so the Secret silently overrides the chart. Editing those keys in values-*.yaml is a
no-op with no warning.

Background / Context

Found while fixing the Vertex embedding 404 (locations/eu does not serve text-embedding-005).
The fix sets a region in values-*.yaml, which would have done nothing: VERTEX_AI_LOCATION was
also projected from prd-rhesis-vertex-ai-location.

#2419 removes VERTEX_AI_LOCATION and VERTEX_AI_PROJECT from all three clusters'
ExternalSecret. That pair was safe to remove because the ConfigMap value provably matched what
production was actually using (the 404 reported projects/playground-437609/locations/eu/...,
identical to the chart). The remaining keys could not be verified without reading Secret Manager, so
they were left in place.

Affected envFrom order (all 7 deployments): charts/rhesis/templates/{backend,worker,frontend,chatbot,polyphemus,otel-collector,telemetry-processor}/deployment.yaml

Remaining duplicated keys

Reproduce:

env=prd
helm template rhesis charts/rhesis -f charts/rhesis/values-$env.yaml \
  | awk '/kind: ConfigMap/,/^---/' \
  | grep -oE '^  [A-Z][A-Z0-9_]+:' | tr -d ' :' | sort -u > /tmp/cm.txt
grep -oE 'secretKey: [A-Z][A-Z0-9_]+' \
  kubernetes/clusters/$env/external-secrets/rhesis-app-secrets.yaml \
  | awk '{print $2}' | sort -u > /tmp/sec.txt
comm -12 /tmp/cm.txt /tmp/sec.txt

Goals

• values-*.yaml is the single source of truth for non-secret config.
• A new overlap cannot be introduced without CI failing.

Steps / Action Plan

1. For each remaining key, read the Secret Manager value and compare it to the chart value.
2. Where they match, delete the key from the ExternalSecret (no behaviour change).
3. Where they differ, decide which is authoritative, align the chart, then delete the duplicate.
4. Add a lint check that intersects rendered ConfigMap keys with ExternalSecret secretKeys and
   fails on any overlap. .github/workflows/lint.yml is the natural home.

Acceptance Criteria (testable)

• No key appears in both the rendered ConfigMap and the ExternalSecret, for dev, stg and prd.
• Secrets stay in the ExternalSecret; non-secret config lives only in the ConfigMap.
• CI fails when a duplicate key is introduced.
• The reproduce command above prints nothing for all three envs.

Risks / Considerations

DB_HOST, DB_NAME, APP_DB_USER and the ANALYTICS_DB_* keys point at real databases. If the
Secret value differs from the chart value, removing the Secret key repoints a live connection. Verify
each value before deleting anything, and do not batch this with unrelated changes.