acoustic-carpenter-78188
04/30/2026, 3:07 PMflytekitplugins-spark) with OAuth Machine-to-Machine (M2M) and OIDC Workload Identity Federation authentication, the two non-legacy authentication paths Databricks now recommends for service-to-service traffic. The connector currently only supports Personal Access Tokens (PATs).
Motivation
Databricks has marked PAT a legacy auth method
Databricks now classifies workspace-level PATs as legacy in their official documentation:
"Where possible, Databricks recommends using OAuth instead of PATs for user account authentication because OAuth provides stronger security." Source: Authenticate with Databricks personal access tokens (legacy)Operational implications of staying on PAT: • 90-day auto-revocation: "Databricks automatically revokes PATs that haven't been used for 90 days." • Per-workspace cap: "A user can create up to 600 PATs per workspace." • No central rotation: PATs can't be rotated through an identity provider; every secret is a separate operational chore. • Weaker auditability: PATs are tied to a single workspace user/SP, with no IdP-level lineage on each call. Goal of this issue Bring the Databricks connector into the modern Databricks auth posture (OAuth M2M (client-credentials) and OIDC Workload Identity Federation (token exchange)) while preserving everything that the prior multi-tenant PAT work in flyteorg/flytekit#3394 delivered: • Per-namespace identity isolation (each Flyte workflow project federates as a different Databricks Service Principal, enabling per-project Unity Catalog access controls). • Backwards-compatible fallback to existing PAT setups. • Zero workflow-code changes for adoption: operators flip auth modes in connector config, workflow authors don't change a line. Related prior work • flyteorg/flyte#6911: Databricks Serverless Compute support (separate track, also in flight). • flyteorg/flytekit#3394: Multi-tenant Databricks PAT via cross-namespace K8s secrets (merged Mar 10, 2026). This PR is the multi-tenancy baseline that the new auth modes preserve. • flyteorg/flytekit#3392: Databricks Serverless support (merged earlier, related task-config surface). Proposed Changes 1. Auth strategy abstraction Introduce a small strategy module (
databricks_auth.py) inside the spark plugin that owns:
• Resolution of the active auth type from connector env vars or per-task config (with auto-detection when unset).
• Token acquisition for each strategy (PAT, OAuth M2M, OIDC Model 1, OIDC Model 2).
• Async-safe in-memory token cache with TTL and a pre-expiry refresh buffer.
• Exponential backoff with jitter on token-endpoint calls.
The connector continues to call a single get_header(...) boundary; the strategy underneath is replaceable.
2. Auth modes
| Auth mode | Credentials | Multi-tenancy granularity | Fallback target |
| --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------- |
| PAT (existing) | K8s Secret databricks-token in workflow namespace | Per-namespace (per [#3394](https://github.com/flyteorg/flyte/pull/3394)) | FLYTE_DATABRICKS_ACCESS_TOKEN env var |
| OAuth M2M | K8s Secret databricks-oauth (client_id + client_secret) in workflow namespace | Per-namespace | Connector env vars FLYTE_DATABRICKS_CLIENT_ID / FLYTE_DATABRICKS_CLIENT_SECRET |
| OIDC Model 1 (Connector-pod identity) | Connector pod's IRSA-projected JWT + connector-level DATABRICKS_CLIENT_ID | Single shared Databricks SP across all workflows | n/a |
| OIDC Model 2 (Per-namespace ServiceAccount) | K8s TokenRequest minted for an annotated SA in the workflow namespace; SA carries flyte.org/databricks-client-id annotation | Per-namespace (each namespace can federate to a distinct Databricks SP) | OIDC Model 1 if no annotated SA found and Model 1 is configured; otherwise fail loudly |
3. Auto-detection order
If FLYTE_DATABRICKS_AUTH_TYPE is unset, the connector picks the strongest reachable mode at submit time, in this order:
1. OIDC if Model 2 SA discoverable in workflow namespace, OR Model 1 prerequisites are met.
2. M2M if `client_id`/`client_secret` reachable (namespace secret or connector env).
3. PAT as the final fallback for backwards compatibility.
If FLYTE_DATABRICKS_AUTH_TYPE is set explicitly, the connector uses that mode and errors loudly when its prerequisites are missing; no silent identity downgrade.
4. OIDC Model 2 discovery (per-namespace tenancy without per-task config)
Operators label/annotate ServiceAccounts in workflow namespaces:
apiVersion: v1
kind: ServiceAccount
metadata:
name: my-project-databricks
namespace: my-project-namespace
labels:
flyte.org/databricks-enabled: "true"
annotations:
flyte.org/databricks-client-id: "00000000-0000-0000-0000-000000000000" # Databricks SP application ID
flyte.org/databricks-audience: "https://my-account.cloud.databricks.com/oidc/v1/token" # optional
At submit time, the connector lists SAs in the workflow's namespace by label, picks the one carrying the flyte.org/databricks-client-id annotation, and mints a JWT for it via the Kubernetes TokenRequest API. The result is exchanged at the Databricks token endpoint for a workspace access token. Workflow authors write no extra config. Different namespaces can federate to different Databricks SPs in the same connector deployment.
Connector RBAC for Model 2 (`get`/`list` on serviceaccounts, plus create on serviceaccounts/token) is documented in the README.
5. Backwards compatibility
• DatabricksJobMetadata and DatabricksV2 task config get additive fields only.
• Existing PAT deployments continue to work unchanged.
• Older DatabricksJobMetadata payloads (without the new fields) are still consumed correctly by upgraded connectors.
API Examples
Operators flip auth modes via connector env vars (workflow code unchanged)
# PAT (legacy, default; unchanged behaviour from #3394)
# (no extra env vars needed; works as today)
# OAuth M2M
FLYTE_DATABRICKS_AUTH_TYPE=oauth_m2m
FLYTE_DATABRICKS_CLIENT_ID=<sp-application-id> # connector-level fallback
FLYTE_DATABRICKS_CLIENT_SECRET=<sp-secret> # connector-level fallback
# Plus per-namespace K8s secret databricks-oauth for tenant overrides.
# OIDC Model 1 (connector-pod identity, shared)
FLYTE_DATABRICKS_AUTH_TYPE=oidc_federation
FLYTE_DATABRICKS_CLIENT_ID=<connector-pod-sp-id>
FLYTE_DATABRICKS_OIDC_AUDIENCE=https://...
# OIDC Model 2 (per-na…
flyteorg/flyte