<#7319 [Core feature] Add OAuth M2M and OIDC Workl...
# flyte-github
a
#7319 [Core feature] Add OAuth M2M and OIDC Workload Identity Federation Authentication Support to Databricks Connector Issue created by rohitrsh Motivation: Why do you think this is important? Extend the Databricks connector (
flytekitplugins-spark
) with OAuth Machine-to-Machine (M2M) and OIDC Workload Identity Federation authentication, the two non-legacy authentication paths Databricks now recommends for service-to-service traffic. The connector currently only supports Personal Access Tokens (PATs). Motivation Databricks has marked PAT a legacy auth method Databricks now classifies workspace-level PATs as legacy in their official documentation:
"Where possible, Databricks recommends using OAuth instead of PATs for user account authentication because OAuth provides stronger security." Source: Authenticate with Databricks personal access tokens (legacy)
Operational implications of staying on PAT: • 90-day auto-revocation: "Databricks automatically revokes PATs that haven't been used for 90 days." • Per-workspace cap: "A user can create up to 600 PATs per workspace." • No central rotation: PATs can't be rotated through an identity provider; every secret is a separate operational chore. • Weaker auditability: PATs are tied to a single workspace user/SP, with no IdP-level lineage on each call. Goal of this issue Bring the Databricks connector into the modern Databricks auth posture (OAuth M2M (client-credentials) and OIDC Workload Identity Federation (token exchange)) while preserving everything that the prior multi-tenant PAT work in flyteorg/flytekit#3394 delivered: • Per-namespace identity isolation (each Flyte workflow project federates as a different Databricks Service Principal, enabling per-project Unity Catalog access controls). • Backwards-compatible fallback to existing PAT setups. • Zero workflow-code changes for adoption: operators flip auth modes in connector config, workflow authors don't change a line. Related prior work • flyteorg/flyte#6911: Databricks Serverless Compute support (separate track, also in flight). • flyteorg/flytekit#3394: Multi-tenant Databricks PAT via cross-namespace K8s secrets (merged Mar 10, 2026). This PR is the multi-tenancy baseline that the new auth modes preserve. • flyteorg/flytekit#3392: Databricks Serverless support (merged earlier, related task-config surface). Proposed Changes 1. Auth strategy abstraction Introduce a small strategy module (
databricks_auth.py
) inside the spark plugin that owns: • Resolution of the active auth type from connector env vars or per-task config (with auto-detection when unset). • Token acquisition for each strategy (PAT, OAuth M2M, OIDC Model 1, OIDC Model 2). • Async-safe in-memory token cache with TTL and a pre-expiry refresh buffer. • Exponential backoff with jitter on token-endpoint calls. The connector continues to call a single
get_header(...)
boundary; the strategy underneath is replaceable. 2. Auth modes | Auth mode | Credentials | Multi-tenancy granularity | Fallback target | | --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------- | | PAT (existing) | K8s Secret databricks-token in workflow namespace | Per-namespace (per [#3394](https://github.com/flyteorg/flyte/pull/3394)) | FLYTE_DATABRICKS_ACCESS_TOKEN env var | | OAuth M2M | K8s Secret databricks-oauth (client_id + client_secret) in workflow namespace | Per-namespace | Connector env vars FLYTE_DATABRICKS_CLIENT_ID / FLYTE_DATABRICKS_CLIENT_SECRET | | OIDC Model 1 (Connector-pod identity) | Connector pod's IRSA-projected JWT + connector-level DATABRICKS_CLIENT_ID | Single shared Databricks SP across all workflows | n/a | | OIDC Model 2 (Per-namespace ServiceAccount) | K8s TokenRequest minted for an annotated SA in the workflow namespace; SA carries flyte.org/databricks-client-id annotation | Per-namespace (each namespace can federate to a distinct Databricks SP) | OIDC Model 1 if no annotated SA found and Model 1 is configured; otherwise fail loudly | 3. Auto-detection order If
FLYTE_DATABRICKS_AUTH_TYPE
is unset, the connector picks the strongest reachable mode at submit time, in this order: 1. OIDC if Model 2 SA discoverable in workflow namespace, OR Model 1 prerequisites are met. 2. M2M if `client_id`/`client_secret` reachable (namespace secret or connector env). 3. PAT as the final fallback for backwards compatibility. If
FLYTE_DATABRICKS_AUTH_TYPE
is set explicitly, the connector uses that mode and errors loudly when its prerequisites are missing; no silent identity downgrade. 4. OIDC Model 2 discovery (per-namespace tenancy without per-task config) Operators label/annotate ServiceAccounts in workflow namespaces: apiVersion: v1 kind: ServiceAccount metadata: name: my-project-databricks namespace: my-project-namespace labels: flyte.org/databricks-enabled: "true" annotations: flyte.org/databricks-client-id: "00000000-0000-0000-0000-000000000000" # Databricks SP application ID flyte.org/databricks-audience: "https://my-account.cloud.databricks.com/oidc/v1/token" # optional At submit time, the connector lists SAs in the workflow's namespace by label, picks the one carrying the
flyte.org/databricks-client-id
annotation, and mints a JWT for it via the Kubernetes
TokenRequest
API. The result is exchanged at the Databricks token endpoint for a workspace access token. Workflow authors write no extra config. Different namespaces can federate to different Databricks SPs in the same connector deployment. Connector RBAC for Model 2 (`get`/`list` on
serviceaccounts
, plus
create
on
serviceaccounts/token
) is documented in the README. 5. Backwards compatibility •
DatabricksJobMetadata
and
DatabricksV2
task config get additive fields only. • Existing PAT deployments continue to work unchanged. • Older
DatabricksJobMetadata
payloads (without the new fields) are still consumed correctly by upgraded connectors. API Examples Operators flip auth modes via connector env vars (workflow code unchanged) # PAT (legacy, default; unchanged behaviour from #3394) # (no extra env vars needed; works as today) # OAuth M2M FLYTE_DATABRICKS_AUTH_TYPE=oauth_m2m FLYTE_DATABRICKS_CLIENT_ID=<sp-application-id> # connector-level fallback FLYTE_DATABRICKS_CLIENT_SECRET=<sp-secret> # connector-level fallback # Plus per-namespace K8s secret
databricks-oauth
for tenant overrides. # OIDC Model 1 (connector-pod identity, shared) FLYTE_DATABRICKS_AUTH_TYPE=oidc_federation FLYTE_DATABRICKS_CLIENT_ID=<connector-pod-sp-id> FLYTE_DATABRICKS_OIDC_AUDIENCE=https://... # OIDC Model 2 (per-na… flyteorg/flyte