Skip to content

setup.md

Latest commit

 

History

History
234 lines (173 loc) · 8.17 KB

setup.md

File metadata and controls

234 lines (173 loc) · 8.17 KB
Raw

Cluster Setup

End-to-end setup for a fresh GKE cluster: Envoy Gateway as the ingress data plane, cert-manager issuing Let's Encrypt certs through the Gateway API, and Dex deployed at https://identity.example.com.

Namespace layout:

  • envoy-gateway-system — Envoy Gateway controller (installed by Helm).
  • cert-manager — cert-manager controller, webhook, cainjector.
  • gateway — the Gateway resource and the TLS secrets cert-manager issues for its listeners.
  • identity — Dex and its HTTPRoute.

Prerequisites

  • kubectl context pointed at the target cluster.
  • helm installed.
  • DNS record for identity.example.com pointing at the Envoy Gateway's external IP. The IP only exists after step 3, so create the record then.

1. Install Envoy Gateway

Reference: https://gateway.envoyproxy.io/docs/tasks/security/tls-cert-manager/

helm install eg oci://docker.io/envoyproxy/gateway-helm \
  --version v1.7.2 \
  -n envoy-gateway-system --create-namespace

We deliberately skip the upstream quickstart.yaml — it creates a GatewayClass, a Gateway in default, and a demo backend. We ship our own GatewayClass and Gateway in gateway.yaml so nothing of ours lands in default.

2. Install cert-manager

crds.enabled=true is required — without it, the post-install startupapicheck job times out because the CRDs are missing. config.enableGatewayAPI=true turns on the Gateway API integration so cert-manager can issue certs for Gateway listeners.

helm repo add jetstack https://charts.jetstack.io

helm install \
  cert-manager jetstack/cert-manager \
  --version v1.17.0 \
  --create-namespace --namespace cert-manager \
  --set crds.enabled=true \
  --set config.apiVersion="controller.config.cert-manager.io/v1alpha1" \
  --set config.kind="ControllerConfiguration" \
  --set config.enableGatewayAPI=true

3. Apply the Gateway

gateway.yaml creates:

  • The gateway namespace.
  • A GatewayClass named eg bound to the Envoy Gateway controller.
  • A Gateway named eg in the gateway namespace with two listeners:
    • HTTP on :80 — required for the ACME HTTP-01 challenge.
    • HTTPS on :443 for identity.example.com, terminating TLS using the secret identity-example-com-tls.
  • The cert-manager.io/cluster-issuer: letsencrypt-prod annotation, so cert-manager auto-creates a Certificate for every TLS listener and writes the cert into the secret named in certificateRefs.
  • allowedRoutes.namespaces.from: All on both listeners, so per-app HTTPRoutes can live with their app rather than centrally.
kubectl apply -f gateway.yaml

Wire up DNS

Once the Gateway is Programmed=True, grab its external IP and create the A record for identity.example.com:

kubectl get gateway eg -n gateway \
  -o jsonpath='{.status.addresses[0].value}'

4. Create the Let's Encrypt ClusterIssuer

cluster-issuer.yaml defines a letsencrypt-prod ClusterIssuer that solves HTTP-01 challenges by attaching a temporary HTTPRoute to the eg Gateway in the gateway namespace.

kubectl apply -f cluster-issuer.yaml

Update the email: field before applying if reusing this for a different project — Let's Encrypt sends expiry notifications to it.

This step has to come after step 3: cert-manager only starts issuing once both the Gateway (with its cluster-issuer annotation) and the ClusterIssuer exist, but the ClusterIssuer's HTTP-01 solver references the Gateway, so the Gateway must already be there.

5. Deploy Dex

dex.yaml bundles everything for the identity service:

  • identity Namespace
  • HTTPRoute binding identity.example.com → the dex Service via the identity-https listener on the Gateway in gateway
  • Dex Secret (config), Deployment, Service, ServiceAccount
kubectl apply -f dex.yaml

The Secret ships a development static-password setup. Replace staticPasswords and staticClients before exposing this to anything that matters.

6. Verify

# Certificate (auto-created by cert-manager from the Gateway annotation)
# should reach Ready=True within ~1–2 minutes.
kubectl get certificate -n gateway -w

# Gateway listeners should report Programmed=True.
kubectl get gateway eg -n gateway -o yaml | yq '.status.listeners'

# Dex should respond with its discovery document over HTTPS.
curl -fsS https://identity.example.com/.well-known/openid-configuration | jq .

7. Deploy the polarsignals-gcp chart

Assumes the GCP infrastructure (buckets, AlloyDB, IAM bindings) has already been provisioned via the polarsignals-gcp terraform module per polarsignals/deploy/public-byoc/GCP.md.

7a. Add Gateway listeners and DNS for the cloud hostnames

gateway.yaml carries HTTPS listeners for the three public cloud endpoints in addition to identity:

  • cloud.example.com (main UI)
  • api.cloud.example.com (REST API)
  • grpc.cloud.example.com (gRPC API)

Apply (or re-apply) the Gateway, then point all three hostnames at the Gateway's external IP via DNS A records:

kubectl apply -f gateway.yaml

kubectl get gateway eg -n gateway \
  -o jsonpath='{.status.addresses[0].value}'

cert-manager will issue a Let's Encrypt cert per listener as soon as DNS resolves. Watch progress:

kubectl get certificate -n gateway -w

7b. Install the Prometheus Operator and Pyrra CRDs

The chart renders ServiceMonitor, PrometheusRule, and ServiceLevelObjective resources. We install only the CRDs so kubectl apply of generated.yaml succeeds — no controllers required:

# monitoring.coreos.com/v1 (ServiceMonitor, PrometheusRule)
kubectl apply --server-side -f \
  https://github.com/prometheus-operator/prometheus-operator/releases/latest/download/stripped-down-crds.yaml

# pyrra.dev/v1alpha1 (ServiceLevelObjective)
kubectl apply --server-side -f \
  https://raw.githubusercontent.com/pyrra-dev/pyrra/main/examples/kubernetes/manifests/setup/pyrra-slo-CustomResourceDefinition.yaml

The CRDs alone make the kubectl apply validate; nothing scrapes the metrics or evaluates the rules until you bring up the matching controllers.

7c. Apply everything

The chart's release namespace is polarsignals-byoc (set in the Makefile). Three Secrets must exist there before applying generated.yaml:

File Contents
db-secret.yaml polarsignals-db — Postgres URL for the AlloyDB proxy sidecar.
token-secret.yaml polarsignals-token-signing-key — opaque signing key.
oidc-secret.yaml polarsignals-oidc — {clientID, clientSecret} matching the Dex private-client.

values.yaml sets global.oidc.issuerURL to the Dex URL, so both --oidc.issuer-url and --sharing-oidc.issuer-url in the rendered manifests point at Dex.

polarsignals-routes.yaml wires each hostname to the matching chart Service:

  • cloud.example.com → Service/main-ui (HTTPRoute)
  • api.cloud.example.com → Service/api port http (HTTPRoute)
  • grpc.cloud.example.com → Service/api port grpc (GRPCRoute)

make deploy applies all of the above (gateway, issuer, dex, namespace, secrets, the rendered chart, and the routes) in dependency order. It does not re-render the chart — run make generate first if values.yaml changed.

make generate   # only if values.yaml changed
make deploy

The sharing endpoints (sharing.cloud.example.com, api.sharing.cloud.example.com, grpc.sharing.cloud.example.com) are not wired up yet — add three more listeners + routes following the same pattern when you need them.

Adding another HTTPS-exposed service

Follow the same per-app pattern Dex uses:

  1. Add a new TLS listener to gateway.yaml with a unique name, the new hostname, and a fresh certificateRefs[0].name.
  2. Apply the updated gateway.yaml. cert-manager will issue a new cert into the named secret in the gateway namespace.
  3. In the app's namespace, create an HTTPRoute whose parentRefs points at eg / gateway with sectionName matching the listener name.