From ee13bc5f2a1c75fbaf18086043e632dfb9964163 Mon Sep 17 00:00:00 2001 From: Michael Hotan Date: Tue, 21 Apr 2026 04:19:09 +1000 Subject: [PATCH] Update selfhosted auth docs for Entra ID and new auth block pattern - Replace deprecated globals (OIDC_BASE_URL, OIDC_CLIENT_ID, CLI_CLIENT_ID) with adminServer.auth block configuration - Add provider-specific tabs: Okta, Entra ID, Generic OIDC - Document Entra ID specifics: scope separation (/.default vs /all), SP Object ID as sub claim, idtyp optional claim, AADSTS errors - Add subjectClaimNames and identityTypeClaimsForApps configuration - Update authorization.md: type "Union" is now preferred over "UserClouds" - Add provider note about sub claim differences in authz service accounts - Add Entra-specific troubleshooting sections - Remove TODO comment Co-Authored-By: Claude Opus 4.6 (1M context) --- .../selfhosted-deployment/authentication.md | 293 +++++++++++++++++- .../selfhosted-deployment/authorization.md | 10 +- 2 files changed, 286 insertions(+), 17 deletions(-) diff --git a/content/deployment/selfhosted-deployment/authentication.md b/content/deployment/selfhosted-deployment/authentication.md index 345c4668a..7a620d71f 100644 --- a/content/deployment/selfhosted-deployment/authentication.md +++ b/content/deployment/selfhosted-deployment/authentication.md @@ -6,8 +6,6 @@ variants: -flyte -serverless -byoc +selfmanaged # Authentication - - {{< key product_name >}} self-hosted deployments use [OpenID Connect (OIDC)](https://openid.net/specs/openid-connect-core-1_0.html) for user authentication and [OAuth 2.0](https://tools.ietf.org/html/rfc6749) for service-to-service authorization. Unlike serverless and BYOC deployments where {{< key product_name >}} manages authentication for you, **self-hosted deployments require you to create and manage OAuth applications in your own identity provider** (e.g. Okta, Microsoft Entra ID, Google Workspace, or any OIDC-compliant provider). {{< key product_name >}} does not provision or manage these applications — you are responsible for their lifecycle, credential rotation, and access policies. @@ -33,23 +31,122 @@ You must use an OIDC-compliant identity provider that you manage outside of {{< Your identity provider must support: -1. **OpenID Connect Discovery** — `/.well-known/openid-configuration` endpoint +1. **OpenID Connect Discovery** — `/.well-known/openid-configuration` or `/.well-known/oauth-authorization-server` endpoint 2. **Authorization Code flow** — for browser and CLI login 3. **Client Credentials flow** — for service-to-service tokens 4. **PKCE** (Proof Key for Code Exchange) — for the CLI public client 5. **Custom scopes** — ability to create an `all` scope (or equivalent) -6. **Custom claims** — ability to add `sub`, `identitytype`, and `preferred_username` claims to access tokens +6. **Custom claims** — ability to emit `sub` and `preferred_username` claims in access tokens. An identity type claim (`identitytype` or equivalent) is recommended for authorization. ### Authorization server setup -Create a custom authorization server (or equivalent) in your identity provider with the following configuration: +Create a custom authorization server (or equivalent) in your identity provider. The setup differs by provider: + +{{< tabs >}} +{{< tab "Okta" >}} +{{< markdown >}} +Create a **Custom Authorization Server** in Okta: - **Audience**: `https://` (the control plane ingress domain) - **Default scope**: `all` -- **Claims**: - - `sub` — set to the user's internal ID for user tokens, or the app's client ID for app tokens - - `identitytype` — set to `"user"` for identity tokens, and `"user"` or `"app"` for access tokens depending on whether the token represents a user or application - - `preferred_username` — set to the user's login for user tokens, or the app's client ID for app tokens (required for identity injection) +- **Metadata URL**: `.well-known/oauth-authorization-server` (Okta-specific, not the standard `openid-configuration`) +- **Claims** (add as access token claims): + - `sub` — Okta populates this natively. For client_credentials tokens, `sub` equals the app's Client ID. + - `identitytype` — set to `"user"` for user tokens, `"app"` for client_credentials tokens + - `preferred_username` — set to the user's login for user tokens, or the app's Client ID for app tokens +{{< /markdown >}} +{{< /tab >}} +{{< tab "Entra ID" >}} +{{< markdown >}} +Register an **App Registration** in Microsoft Entra ID: + +- **App ID URI**: `api://` (this becomes the audience) +- **Metadata URL**: `.well-known/openid-configuration` (standard OIDC) +- **Scopes**: Create two custom scopes on the app registration: + - `all` — delegated scope for browser login (authorization_code flow) + - `/.default` — used automatically for client_credentials flow +- **Claims** — configure via the app manifest's `optionalClaims`: + - `sub` — Entra populates this natively. For client_credentials tokens, `sub` equals the **Service Principal Object ID** (not the Client ID). + - `idtyp` — add as an optional access token claim. Emits `"app"` for client_credentials tokens (maps to the `identitytype` concept). + - `preferred_username` — included by default for user tokens + +> [!WARNING] +> Entra ID uses `sub` = Service Principal Object ID for client_credentials tokens, not the Client ID. When configuring trusted identities for service-to-service auth, use the SP Object ID (found in Enterprise Applications, not App Registrations). + +> [!NOTE] +> Entra ID requires different scopes for different flows: +> - **Browser login** (authorization_code): `api:///all` — Entra rejects `/.default` for same-app auth_code flows (error `AADSTS90009`) +> - **CLI** (authorization_code + PKCE): `api:///.default` +> - **Service-to-service** (client_credentials): `api:///.default` +{{< /markdown >}} +{{< /tab >}} +{{< tab "Generic OIDC" >}} +{{< markdown >}} +For other OIDC providers (Keycloak, Authentik, Auth0, etc.): + +- **Audience**: `https://` or a custom resource identifier +- **Metadata URL**: Usually `.well-known/openid-configuration` +- **Scopes**: Create an `all` scope (or use your provider's default scope) +- **Claims**: Ensure access tokens include: + - `sub` — a stable identifier for the authenticated principal + - `preferred_username` — display name for identity injection + - An identity type claim is optional but recommended for authorization + +If your IdP cannot emit an `identitytype` claim, see the [identity type claim requirements](#identity-type-claim-requirements) section below. + +If your IdP's client_credentials tokens omit the `sub` claim, configure `subjectClaimNames` to specify a fallback chain (e.g., `["sub", "client_id", "azp"]`). +{{< /markdown >}} +{{< /tab >}} +{{< /tabs >}} + +### Identity type claim requirements + +{{< key product_name >}} uses an identity type claim to distinguish human users from service applications. This distinction is **required for Union (built-in RBAC) authorization** and affects how access control decisions are made. + +Your IdP must emit a claim that maps to the `identitytype` concept, with values that distinguish user tokens from application tokens. The claim name and values are configurable: + +| Provider | Claim name | User value | App value | Configuration | +|----------|-----------|------------|-----------|---------------| +| Okta | `identitytype` | `"user"` | `"app"` | Custom access token claim on authorization server | +| Entra ID | `idtyp` | (not emitted) | `"app"` | Enable via optional claims in app manifest. Map with `identityTypeClaimsForApps: {idtyp: ["app"]}` | +| Generic | varies | varies | varies | Configure `identityTypeClaimsForApps` to map your claim name and values | + +> [!WARNING] +> **Union authorization mode requires identity type resolution.** If your IdP cannot emit any claim that distinguishes users from applications, you must either: +> 1. Set `global.USE_EXTERNAL_IDENTITY: true` — the platform will infer identity type from the authentication context (e.g., whether the token was issued via authorization_code or client_credentials flow). This works for basic cases but may not cover all scenarios. +> 2. Use **External authorization mode** instead of Union mode — your external authorization server can determine identity type from the JWT payload, `sub` claim, or any other token attribute directly, without relying on the platform's identity type resolution. +> +> Without identity type resolution, Union authorization cannot distinguish user requests from service account requests, which may result in incorrect access control decisions. + +### Subject claim requirements + +{{< key product_name >}} uses the JWT `sub` claim as the **primary identifier** for all callers — users and service accounts alike. This value is used for: + +- **Authorization decisions** — matching callers to roles and permissions +- **Trusted identity validation** — verifying internal service-to-service callers +- **Audit logging** — recording who performed each action +- **Resource ownership** — the "Owned By" relationship in the console + +> [!WARNING] +> The `sub` claim value must be **stable and unique** per principal. If your IdP returns different `sub` values for the same user across token refreshes, authorization and ownership tracking will break. + +**Your IdP must emit a `sub` claim in all access tokens.** If your IdP's client_credentials tokens use a different claim for the caller identity (or omit `sub` entirely), configure `subjectClaimNames` to specify a fallback chain: + +```yaml +# In flyte.configmap.adminServer.auth.appAuth.externalAuthServer: +subjectClaimNames: + - sub # Standard OIDC subject (tried first) + - client_id # OAuth2 client ID (common fallback) + - azp # Authorized party (alternative) +``` + +The platform tries each claim in order and uses the first non-empty value as the caller's identity. + +> [!NOTE] +> **Provider-specific `sub` values:** +> - **Okta**: `sub` equals the Client ID for client_credentials tokens and the user's Okta ID for user tokens. +> - **Entra ID**: `sub` equals the **Service Principal Object ID** for client_credentials tokens (not the Client ID). Find this in Entra ID > Enterprise Applications > your app > Object ID. +> - When configuring trusted identities for internal services (e.g., `INTERNAL_SUBJECT_ID`), use the value that your IdP places in the `sub` claim — not necessarily the Client ID. ## Step 1: Create OAuth2 applications @@ -116,19 +213,22 @@ Note the **Client ID** and **Client Secret** — these are encoded into the EAGE ## Step 2: Configure control plane -Add OIDC settings to your control plane overrides file: +Authentication is configured in the `flyte.configmap.adminServer.auth` block in your control plane Helm values. This block defines how the admin service validates tokens, which clients are trusted, and how browser login works. + +You also need to set a few global variables for service-to-service authentication: ```yaml global: - OIDC_BASE_URL: "https://your-idp.example.com/oauth2/default" - OIDC_CLIENT_ID: "" # App 1 - CLI_CLIENT_ID: "" # App 2 INTERNAL_CLIENT_ID: "" # App 3 - AUTH_TOKEN_URL: "https://your-idp.example.com/oauth2/default/v1/token" + AUTH_TOKEN_URL: "" # OAuth2 token endpoint + OIDC_S2S_SCOPE: "" # Leave empty for Okta, set to "api:///.default" for Entra ID ``` -Enable authentication in the admin service: +Then configure the auth block. Select your identity provider below: +{{< tabs >}} +{{< tab "Okta" >}} +{{< markdown >}} ```yaml flyte: configmap: @@ -136,11 +236,162 @@ flyte: server: security: useAuth: true + auth: + appAuth: + authServerType: External + externalAuthServer: + baseUrl: "https://dev-123456.okta.com/oauth2/default" + metadataUrl: ".well-known/oauth-authorization-server" + allowedAudience: + - "https://" + thirdPartyConfig: + flyteClient: + clientId: "" # App 2 + redirectUri: "http://localhost:53593/callback" + scopes: + - all + userAuth: + openId: + baseUrl: "https://dev-123456.okta.com/oauth2/default" + clientId: "" # App 1 + scopes: + - profile + - openid + - offline_access + cookieSetting: + sameSitePolicy: LaxMode + domain: "" ``` +Set globals: +```yaml +global: + INTERNAL_CLIENT_ID: "" + AUTH_TOKEN_URL: "https://dev-123456.okta.com/oauth2/default/v1/token" + OIDC_S2S_SCOPE: "" # Okta defaults to "all" +``` +{{< /markdown >}} +{{< /tab >}} +{{< tab "Entra ID" >}} +{{< markdown >}} +```yaml +flyte: + configmap: + adminServer: + server: + security: + useAuth: true + auth: + appAuth: + authServerType: External + externalAuthServer: + baseUrl: "https://login.microsoftonline.com//v2.0" + metadataUrl: ".well-known/openid-configuration" + allowedAudience: + - "api://" + # Entra client_credentials tokens use SP Object ID as sub, not client_id. + # Configure fallback chain: + subjectClaimNames: + - sub + - client_id + # Map Entra's idtyp claim to internal identitytype: + identityTypeClaimsForApps: + idtyp: + - app + thirdPartyConfig: + flyteClient: + clientId: "" # App 2 + redirectUri: "http://localhost:53593/callback" + scopes: + - "api:///.default" + audience: "api://" + userAuth: + openId: + baseUrl: "https://login.microsoftonline.com//v2.0" + clientId: "" # App 1 + scopes: + - profile + - openid + - offline_access + - "api:///all" + cookieSetting: + sameSitePolicy: LaxMode + domain: "" +``` + +Set globals: +```yaml +global: + INTERNAL_CLIENT_ID: "" + AUTH_TOKEN_URL: "https://login.microsoftonline.com//oauth2/v2.0/token" + OIDC_S2S_SCOPE: "api:///.default" +``` + +> [!NOTE] +> `INTERNAL_SUBJECT_ID` defaults to `INTERNAL_CLIENT_ID` for backward compatibility. For Entra ID, where the token `sub` claim is the Service Principal Object ID (not the Client ID), set `INTERNAL_SUBJECT_ID` to the SP Object ID. Find this in **Entra ID > Enterprise Applications > your app > Object ID**. +{{< /markdown >}} +{{< /tab >}} +{{< tab "Generic OIDC" >}} +{{< markdown >}} +```yaml +flyte: + configmap: + adminServer: + server: + security: + useAuth: true + auth: + appAuth: + authServerType: External + externalAuthServer: + baseUrl: "" + metadataUrl: ".well-known/openid-configuration" + allowedAudience: + - "" + thirdPartyConfig: + flyteClient: + clientId: "" # App 2 + redirectUri: "http://localhost:53593/callback" + scopes: + - all + userAuth: + openId: + baseUrl: "" + clientId: "" # App 1 + scopes: + - profile + - openid + - offline_access + cookieSetting: + sameSitePolicy: LaxMode + domain: "" +``` + +Set globals: +```yaml +global: + INTERNAL_CLIENT_ID: "" + AUTH_TOKEN_URL: "/token" # Your IdP's token endpoint + OIDC_S2S_SCOPE: "" # Set if your IdP requires a specific scope for client_credentials +``` + +If your IdP's client_credentials tokens don't include a `sub` claim, add: +```yaml + subjectClaimNames: + - sub + - client_id + - azp +``` +{{< /markdown >}} +{{< /tab >}} +{{< /tabs >}} + > [!NOTE] > Setting `useAuth: true` is required for the `/login`, `/callback`, and `/me` endpoints to register. Without this, auth endpoints will return 404. +> [!NOTE] +> **Deprecated globals:** `OIDC_BASE_URL`, `OIDC_CLIENT_ID`, and `CLI_CLIENT_ID` are deprecated but still functional. New deployments should use the `auth` block directly as shown above. Existing deployments using these globals will continue to work. + ## Step 3: Create Kubernetes secrets (control plane) The control plane needs secrets for the browser login app (App 1) and the service-to-service app (App 3): @@ -299,3 +550,15 @@ Ensure the CLI app (App 2) redirect URIs include `http://localhost:53593/callbac uctl config init --host https:// uctl get project ``` + +### Entra ID: `AADSTS90009` on browser login + +This error occurs when using the `/.default` scope with an authorization_code flow on the same app. Entra ID requires a named delegated scope (e.g., `api:///all`) for browser login. Check that `userAuth.openId.scopes` includes `api:///all` and not `/.default`. + +### Entra ID: `AADSTS1002012` invalid_scope for service-to-service + +Client_credentials flows in Entra ID require the `/.default` scope. Ensure `OIDC_S2S_SCOPE` is set to `api:///.default` in your globals. + +### Subject not found in token + +If flyteadmin logs show `subject claim not found`, your IdP's client_credentials tokens may not include a `sub` claim. Configure `subjectClaimNames` in the auth block to specify a fallback chain (e.g., `["sub", "client_id"]`). diff --git a/content/deployment/selfhosted-deployment/authorization.md b/content/deployment/selfhosted-deployment/authorization.md index f9faddff9..d1bd34478 100644 --- a/content/deployment/selfhosted-deployment/authorization.md +++ b/content/deployment/selfhosted-deployment/authorization.md @@ -83,13 +83,16 @@ No authorization enforcement — all authenticated requests are allowed. This is {{< key product_name >}}'s built-in authorization engine, **embedded in the controlplane Helm chart**. It deploys automatically when enabled, with no separate chart installation required. Provides role-based access control with predefined roles (Admin, Contributor, Viewer) and policy-based fine-grained permissions. > [!NOTE] -> The Helm config value `type: "UserClouds"` is a legacy name from an earlier implementation. It activates {{< key product_name >}}'s built-in authorization engine. This will be renamed to `type: "Union"` in a future release. +> The Helm config accepts both `type: "Union"` (preferred) and `type: "UserClouds"` (legacy name). Both activate the same built-in authorization engine. Use `"Union"` for new deployments. **When to use:** - Production deployments wanting out-of-the-box RBAC with no additional infrastructure - Organizations that need role management through the {{< key product_name >}} console - Teams wanting a performant, battle-tested authorization backend with low operational burden +> [!WARNING] +> **Union mode requires identity type claim resolution.** Your IdP must emit a claim that the platform can map to distinguish user tokens from application tokens (e.g., Okta's `identitytype` or Entra ID's `idtyp`). If your IdP cannot provide this, either set `global.USE_EXTERNAL_IDENTITY: true` or use **External** authorization mode instead. See [Identity type claim requirements]({{< relref "authentication#identity-type-claim-requirements" >}}) in the authentication guide. + **Trade-offs:** - Built-in role management (Admin, Contributor, Viewer) with full RBAC — assign users and groups to roles with resource-level granularity - Zero additional infrastructure — embedded in the controlplane chart, managed by {{< key product_name >}} @@ -209,7 +212,10 @@ Your external authorization server **must** grant appropriate permissions to the ### Configuring service accounts -The three internal OAuth apps must be registered in your external server's permission mapping. Their `sub` claims are the OAuth **client IDs** from your identity provider — the same values configured during [authentication setup]({{< relref "authentication" >}}). +The three internal OAuth apps must be registered in your external server's permission mapping. Their `sub` claims identify the calling application — use the values configured during [authentication setup]({{< relref "authentication" >}}). + +> [!NOTE] +> **Provider-specific subject values:** For Okta, the `sub` claim in client_credentials tokens equals the app's **Client ID**. For Entra ID, the `sub` claim equals the app's **Service Principal Object ID** (found in Entra ID > Enterprise Applications). If you configured `subjectClaimNames` with a fallback chain, the resolved subject may come from a different claim (e.g., `client_id` as fallback). To find the client IDs for your deployment, check the controlplane Helm values: