feat(mcp): always-on HTTP auth via typed build_mcp_auth with Airbyte Cloud defaults

airbyte/mcp/__init__.py

@@ -151,32 +151,36 @@
 credentials that matter are your Airbyte Cloud creds in the dotenv file.
 
 When the server is instead exposed over **HTTP** (`airbyte-mcp-http` /
-`poe mcp-serve-http`), it verifies an `Authorization: Bearer <token>` on every
-request. Two client shapes are supported on the same deployment (combined
-automatically when both are configured):
+`poe mcp-serve-http`), transport auth is **always on** and targets Airbyte Cloud
+out of the box — it verifies an `Authorization: Bearer <token>` on every request
+against Airbyte Cloud's public Keycloak realms with no auth config required. Two
+client shapes are supported on the same deployment (combined automatically):
 
 ### Humans → interactive OIDC
 
-Set `OIDC_CONFIG_URL`, `OIDC_CLIENT_ID`, and `OIDC_CLIENT_SECRET`. Interactive
-clients open a browser (Keycloak Authorization Code + PKCE) and the resulting
-token is verified by the server. No bearer token to manage by hand.
+Supply `AIRBYTE_MCP_OIDC_CLIENT_ID` and `AIRBYTE_MCP_OIDC_CLIENT_SECRET` (the
+OIDC discovery URL defaults to Airbyte Cloud). Interactive clients open a browser
+(Keycloak Authorization Code + PKCE) and the resulting token is verified by the
+server. No bearer token to manage by hand.
 
 ### Machines / agents → headless bearer token
 
-There is **no** transport mode that accepts a raw `client_id` + `client_secret`
-in a header. A headless agent **mints its own short-lived bearer token** and
-sends it as `Authorization: Bearer <token>`; the server verifies the signature
-(no browser, no stored/rotating refresh token).
+By default there is **no** transport mode that accepts a raw `client_id` +
+`client_secret` in a header (see the opt-in exception below). A headless agent
+**mints its own short-lived bearer token** and sends it as
+`Authorization: Bearer <token>`; the server verifies the signature (no browser,
+no stored/rotating refresh token).
 
-For Airbyte Cloud, set `MCP_AUTH_AIRBYTE_CLOUD=true` on the server so it verifies
-tokens against Airbyte Cloud's application-client realm without hand-configuring
-URLs. The agent mints an Airbyte Cloud access token from its
+By default the server verifies against Airbyte Cloud's application-client realm,
+so an agent just mints an Airbyte Cloud access token from its
 `AIRBYTE_CLOUD_CLIENT_ID` / `AIRBYTE_CLOUD_CLIENT_SECRET` (the
 `https://api.airbyte.com/v1/applications/token` endpoint) and sends it as the
 bearer. That single token both authenticates transport (verified by the server)
 and authorizes downstream Cloud API calls, because an Airbyte-Cloud-issued token
 is itself a valid Cloud API bearer. Tokens are short-lived (~15 min), so
-re-mint on expiry / on a `401` rather than pinning a static token.
+re-mint on expiry / on a `401` rather than pinning a static token. A self-hosted
+deployment pointing at its own Airbyte instance overrides the realm via the
+`AIRBYTE_MCP_AUTH_*` vars below.
 
 Clients that support HTTP transports can pass the token via a `headers` block in
 their MCP config:
@@ -194,24 +198,61 @@
 }
 ```
 
+### Opt-in: static client credentials via HTTP Basic
+
+The headless bearer path assumes the client can re-mint a token every ~15 min.
+An agent that can only set a **static** `Authorization` value (no refresh hook)
+can't do that. For this case the server can optionally accept the long-lived
+`client_id` / `client_secret` directly on the inbound request, via standard HTTP
+Basic (the same credential encoding OAuth's `client_secret_basic` uses):
+
+```http
+Authorization: Basic base64(client_id:client_secret)
+```
+
+When `AIRBYTE_MCP_AUTH_ALLOW_CLIENT_CREDENTIALS` is truthy, the server runs a
+client-credentials exchange server-side (posting the credentials to the Airbyte
+token endpoint) for a short-lived token, caches it until shortly before expiry,
+and verifies it via the same headless bearer path — so the agent presents one
+durable credential and the server owns the token churn. This is **off by
+default**, because accepting long-lived credentials at the transport is a
+deliberate escalation; a `Bearer` request is always preferred when the client
+can manage it.
+
 ### Server environment variables (HTTP mode)
 
+All auth settings default to Airbyte Cloud's public (non-secret) Keycloak realms,
+so the hosted Airbyte Cloud deployment needs none of them except the interactive
+OIDC client credentials. Each is an override for a self-hosted deployment that
+points at its own Airbyte instance:
+
 - `MCP_SERVER_URL` — public base URL of the server (also used for OIDC redirect
   callbacks); defaults to `http://localhost:8080`.
-- `OIDC_CONFIG_URL`, `OIDC_CLIENT_ID`, `OIDC_CLIENT_SECRET` — enable interactive
-  OIDC (all three required).
-- `MCP_AUTH_AIRBYTE_CLOUD` — set truthy to verify headless tokens against Airbyte
-  Cloud's realm (fills JWKS URI / issuer / audience / algorithm with Airbyte
-  Cloud defaults; each stays overridable).
-- `MCP_AUTH_JWKS_URI` / `MCP_AUTH_JWT_PUBLIC_KEY` — JWKS URL or static public key
-  for verifying headless tokens.
-- `MCP_AUTH_ISSUER` / `MCP_AUTH_AUDIENCE` / `MCP_AUTH_ALGORITHM` — expected `iss`
-  / `aud` claims and signing algorithm (recommended for headless).
-
-If no transport auth variables are set, the HTTP server starts
-**unauthenticated** and logs a warning — acceptable only behind a trusted
-network boundary. The verifier assembly is provided by
-[`fastmcp-extensions`](https://github.com/airbytehq/fastmcp-extensions).
+- `AIRBYTE_MCP_OIDC_CLIENT_ID` / `AIRBYTE_MCP_OIDC_CLIENT_SECRET` — OAuth client
+  credentials that activate the interactive OIDC path (no default; deployment
+  supplies them).
+- `AIRBYTE_MCP_OIDC_CONFIG_URL` — OIDC discovery URL; defaults to Airbyte Cloud's
+  `airbyte` realm.
+- `AIRBYTE_MCP_AUTH_JWKS_URI` — JWKS URL for verifying headless bearer tokens;
+  defaults to Airbyte Cloud's application-client realm certs.
+- `AIRBYTE_MCP_AUTH_JWT_PUBLIC_KEY` — static public key for verifying headless
+  bearer tokens; an alternative to `AIRBYTE_MCP_AUTH_JWKS_URI` for self-hosted
+  realms without a JWKS endpoint (no default).
+- `AIRBYTE_MCP_AUTH_ISSUER` / `AIRBYTE_MCP_AUTH_AUDIENCE` /
+  `AIRBYTE_MCP_AUTH_ALGORITHM` — expected `iss` / `aud` claims and signing
+  algorithm; default to Airbyte Cloud (`.../realms/_airbyte-application-clients`,
+  `account`, `RS256`).
+- `AIRBYTE_MCP_AUTH_ALLOW_CLIENT_CREDENTIALS` — opt-in flag (off by default) that
+  enables the inbound HTTP Basic client-credentials path described above.
+- `AIRBYTE_MCP_AUTH_CLIENT_CREDENTIALS_TOKEN_URL` — token endpoint used for that
+  exchange; defaults to Airbyte Cloud's `https://api.airbyte.com/v1/applications/token`.
+
+This module owns those Airbyte-branded names, reads them itself, and maps them
+into the typed `JWTAuthConfig` / `OIDCAuthConfig` objects it passes to the
+`build_mcp_auth` function from
+[`fastmcp-extensions`](https://github.com/airbytehq/fastmcp-extensions), which
+assembles the verifiers. The extensions library reads no environment variables
+of its own, keeping it provider-agnostic.
 
 ## Troubleshooting
 

airbyte/mcp/_client_credentials.py

@@ -0,0 +1,81 @@
+# Copyright (c) 2025 Airbyte, Inc., all rights reserved.
+"""Opt-in HTTP Basic client-credentials transport auth for the MCP server.
+
+The headless bearer path verifies an already-minted, short-lived (~15 min) JWT.
+That works for MCP clients that run the OAuth flow and refresh tokens
+automatically, but not for a truly headless agent that can only set a *static*
+`Authorization` header value and cannot re-mint on a timer.
+
+This module bridges that gap, behind an opt-in flag. When enabled, the server
+accepts the long-lived `client_id` / `client_secret` presented on the inbound MCP
+request via standard HTTP Basic auth
+(`Authorization: Basic base64(client_id:client_secret)`, the same credential
+encoding OAuth's `client_secret_basic` uses). The server then runs a
+client-credentials exchange against the Airbyte token endpoint to obtain a
+short-lived access token and rewrites the request to `Authorization: Bearer
+<token>` so the existing `JWTVerifier` validates it unchanged. The agent thus
+presents a durable credential once; the server owns the short-lived-token churn.
+
+The provider-neutral exchange middleware lives in `fastmcp_extensions`
+(`wrap_client_credentials`). This module owns only the Airbyte-specific policy:
+the opt-in env toggle and the Airbyte Cloud token endpoint (overridable for
+self-hosted deployments). It resolves those to plain values and hands them to
+the generic library, so no Airbyte literal or env-var name leaks into the lib.
+"""
+
+from __future__ import annotations
+
+import os
+from typing import TYPE_CHECKING
+
+from fastmcp_extensions import wrap_client_credentials
+
+
+if TYPE_CHECKING:
+    from collections.abc import Mapping
+
+    from starlette.types import ASGIApp
+
+
+# Opt-in flag. Off by default: accepting long-lived credentials at the transport
+# is a deliberate escalation, so a deployment must explicitly turn it on.
+ALLOW_CLIENT_CREDENTIALS_ENV = "AIRBYTE_MCP_AUTH_ALLOW_CLIENT_CREDENTIALS"
+
+# Airbyte token endpoint that mints an application access token from a
+# `client_id` / `client_secret`. Defaults to Airbyte Cloud; overridable for
+# self-hosted deployments pointing at their own Airbyte instance.
+AIRBYTE_CLOUD_TOKEN_URL = "https://api.airbyte.com/v1/applications/token"
+TOKEN_URL_ENV = "AIRBYTE_MCP_AUTH_CLIENT_CREDENTIALS_TOKEN_URL"
+
+_TRUTHY = frozenset({"1", "true", "t", "yes", "y", "on"})
+
+
+def client_credentials_enabled(env: Mapping[str, str] | None = None) -> bool:
+    """Return whether the opt-in HTTP Basic client-credentials grant is enabled."""
+    source = env if env is not None else os.environ
+    return source.get(ALLOW_CLIENT_CREDENTIALS_ENV, "").strip().lower() in _TRUTHY
+
+
+def _token_url() -> str:
+    """Return the token endpoint, defaulting to Airbyte Cloud.
+
+    A blank or whitespace-only override is treated as unset so the Airbyte Cloud
+    default still applies, rather than POSTing to an invalid URL and failing every
+    Basic-auth request closed.
+    """
+    return os.getenv(TOKEN_URL_ENV, "").strip() or AIRBYTE_CLOUD_TOKEN_URL
+
+
+def wrap_if_enabled(app: ASGIApp) -> ASGIApp:
+    """Wrap `app` with the client-credentials exchange when the flag is set.
+
+    Returns `app` unchanged when the opt-in flag is unset, so the standard
+    bearer/OIDC transport auth is the only path. When enabled, wraps `app` as the
+    outermost ASGI layer (via `fastmcp_extensions.wrap_client_credentials`) so the
+    Basic-to-Bearer rewrite happens before FastMCP's auth verifier runs.
+    """
+    return wrap_client_credentials(
+        app,
+        enabled=client_credentials_enabled(),
+        token_url=_token_url(),
+    )

airbyte/mcp/http_main.py

@@ -2,50 +2,70 @@
 """HTTP transport entry point for the Airbyte MCP server.
 
 Starts the MCP server with HTTP transport, suitable for hosted deployment
-behind a load balancer. Transport auth is assembled in `server.py` via
-`fastmcp_extensions.resolve_mcp_auth`, which supports interactive OIDC and
-headless bearer-token verification (combined via `MultiAuth` when both are
-configured). See `server.py` for details.
+behind a load balancer. Transport auth is assembled in `server.py`, which
+defaults every setting to Airbyte Cloud's public Keycloak realms and supports
+interactive OIDC plus headless bearer-token verification (combined via
+`MultiAuth`). See `server.py` for details.
 
 Environment variables:
 
 - `MCP_SERVER_URL`: Public base URL. Used for OIDC redirect callbacks and to
   derive the MCP endpoint mount path (serves at `/` when the URL has a path
   prefix, otherwise defaults to `/mcp`).
 
-Interactive OIDC (Keycloak Authorization Code + PKCE), enabled when all three
-are set:
+Interactive OIDC (Keycloak Authorization Code + PKCE), active once the client
+credentials are supplied:
 
-- `OIDC_CONFIG_URL`: Keycloak OIDC discovery URL
-- `OIDC_CLIENT_ID`: OIDC client identifier
-- `OIDC_CLIENT_SECRET`: OIDC client secret
+- `AIRBYTE_MCP_OIDC_CLIENT_ID`: OIDC client identifier
+- `AIRBYTE_MCP_OIDC_CLIENT_SECRET`: OIDC client secret
+- `AIRBYTE_MCP_OIDC_CONFIG_URL`: Keycloak OIDC discovery URL (defaults to
+  Airbyte Cloud)
 
 Headless bearer-token verification (for agents/CI that mint their own
-short-lived token via the client credentials grant), enabled when
-`MCP_AUTH_JWKS_URI` or `MCP_AUTH_JWT_PUBLIC_KEY` is set:
-
-- `MCP_AUTH_JWKS_URI`: JWKS endpoint used to verify token signatures
-- `MCP_AUTH_JWT_PUBLIC_KEY`: static public key (alternative to `MCP_AUTH_JWKS_URI`)
-- `MCP_AUTH_ISSUER`: expected token issuer
-- `MCP_AUTH_AUDIENCE`: expected token audience
-- `MCP_AUTH_ALGORITHM`: signing algorithm override
+short-lived token via the client credentials grant); on by default against
+Airbyte Cloud, overridable for self-hosted deployments:
+
+- `AIRBYTE_MCP_AUTH_JWKS_URI`: JWKS endpoint used to verify token signatures
+- `AIRBYTE_MCP_AUTH_JWT_PUBLIC_KEY`: static public key (alternative to
+  `AIRBYTE_MCP_AUTH_JWKS_URI` for realms without a JWKS endpoint)
+- `AIRBYTE_MCP_AUTH_ISSUER`: expected token issuer
+- `AIRBYTE_MCP_AUTH_AUDIENCE`: expected token audience
+- `AIRBYTE_MCP_AUTH_ALGORITHM`: signing algorithm
+
+Opt-in HTTP Basic client-credentials transport auth, for a headless agent that
+can only set a static `Authorization` header and cannot re-mint short-lived
+tokens itself (off by default):
+
+- `AIRBYTE_MCP_AUTH_ALLOW_CLIENT_CREDENTIALS`: when truthy, accept
+  `Authorization: Basic base64(client_id:client_secret)`, exchange it for a
+  short-lived bearer token server-side, and verify that token via the headless
+  path above
+- `AIRBYTE_MCP_AUTH_CLIENT_CREDENTIALS_TOKEN_URL`: token endpoint used for the
+  exchange (defaults to Airbyte Cloud)
+
+Running over HTTP always requires transport auth: startup fails fast if no auth
+provider resolves (e.g. an invalid `AIRBYTE_MCP_AUTH_*` override, since blank
+values fall back to the Airbyte Cloud defaults rather than disabling auth).
 """
 
 from __future__ import annotations
 
 import logging
-import os
 from urllib.parse import urlparse
 
+import uvicorn
 from fastmcp_extensions import (
     assert_http_trusted_execution_disabled,
     register_landing_page,
 )
 
+from airbyte.mcp._client_credentials import wrap_if_enabled
 from airbyte.mcp.server import (
     DEFAULT_HTTP_HOST,
     DEFAULT_HTTP_PORT,
+    DEFAULT_MCP_SERVER_URL,
     MCP_SERVER_URL_ENV,
+    _env_or_default,
     app,
 )
 
@@ -58,11 +78,13 @@
 
 
 def _get_server_url() -> str:
-    """Return the public base URL from `MCP_SERVER_URL`, defaulting to localhost."""
-    return os.getenv(
-        MCP_SERVER_URL_ENV,
-        f"http://localhost:{DEFAULT_HTTP_PORT}",
-    )
+    """Return the public base URL from `MCP_SERVER_URL`, defaulting to localhost.
+
+    A blank or whitespace-only value is treated as unset (matching `_create_auth`
+    in `server.py`), so the mount-path and landing-page URL derived here can't
+    disagree with the OIDC redirect base derived there.
+    """
+    return _env_or_default(MCP_SERVER_URL_ENV, DEFAULT_MCP_SERVER_URL)
 
 
 def main() -> None:
@@ -91,12 +113,16 @@ def main() -> None:
     )
 
     if getattr(app, "auth", None) is None:
-        logger.warning(
-            "HTTP transport starting without authentication: no interactive "
-            "OIDC or headless bearer-token auth is configured, so every "
-            "request is unauthenticated. Set `OIDC_CONFIG_URL`/`OIDC_CLIENT_ID`/"
-            "`OIDC_CLIENT_SECRET` (interactive) or `MCP_AUTH_JWKS_URI`/"
-            "`MCP_AUTH_JWT_PUBLIC_KEY` (headless) to require auth."
+        # HTTP transport is always authenticated by design: launching over HTTP
+        # declares the server needs auth. Auth defaults to Airbyte Cloud and blank
+        # env vars fall back to those defaults, so a `None` provider means auth
+        # resolution produced nothing at all — e.g. an invalid `AIRBYTE_MCP_AUTH_*`
+        # override. Fail fast rather than fall open to unauthenticated serving.
+        raise RuntimeError(
+            "HTTP transport requires transport auth, but none resolved. Auth "
+            "defaults to Airbyte Cloud; leave the `AIRBYTE_MCP_AUTH_*` settings "
+            "unset to use those defaults, or set them to valid values. Check "
+            "startup logs for auth-resolution warnings before serving over HTTP."
         )
 
     logger.info(
@@ -112,12 +138,21 @@ def main() -> None:
     # entrypoint (a permanent gate; the per-request filter also forces it off).
     assert_http_trusted_execution_disabled(app)
 
-    app.run(
+    # Build the ASGI app ourselves (rather than `app.run`) so the optional
+    # client-credentials exchange can wrap it as the *outermost* layer — ahead
+    # of FastMCP's auth middleware — so its Basic-to-Bearer rewrite is what the
+    # verifier sees. When the opt-in flag is unset, `wrap_if_enabled` returns the
+    # app unchanged. The Starlette app owns the session-manager lifespan, so
+    # running it under uvicorn directly is equivalent to `app.run`.
+    http_app = app.http_app(
+        path=mcp_path,
         transport="streamable-http",
+        stateless_http=True,
+    )
+    uvicorn.run(
+        wrap_if_enabled(http_app),
         host=DEFAULT_HTTP_HOST,
         port=DEFAULT_HTTP_PORT,
-        path=mcp_path,
-        stateless_http=True,
     )