Hi Everyone :wave: I just opened a PR that adds O...
# contribute
a
Hi Everyone 👋 I just opened a PR that adds OAuth M2M and OIDC workload-identity federation to the Databricks connector, and refactors the existing PAT support so all four modes share one extension point. Would love eyes from anyone running the Databricks integration today, or anyone who reviewed the multi-tenant PAT work. Why now: Databricks marked PATs as legacy (https://docs.databricks.com/aws/en/dev-tools/auth/pat) and is steering customers toward OAuth M2M + OIDC federation. The connector previously only spoke PAT. What's in the PR:
PATAuth
(refactored from flyteorg/flytekit#3394, behaviour preserved) • `OAuthM2MAuth`: per-namespace
databricks-oauth
k8s secret with env-var fallback •
OIDCConnectorIRSAAuth
(Model 1): connector pod's projected JWT (e.g. EKS IRSA) •
OIDCNamespaceSAAuth
(Model 2): per-workflow-namespace ServiceAccount JWT minted via Kubernetes TokenRequest, discovered from SA labels and annotations. Preserves the per-namespace tenancy story for Unity Catalog. Auto-detection + token refresh on 401 + zero workflow-code changes for adoption (operators flip env vars, task authors don't touch anything). Validation: 100/100 unit tests, end-to-end on a dev EKS cluster against a real Databricks workspace for PAT, M2M, and OIDC Model 2. Links: • Issue: https://github.com/flyteorg/flyte/issues/7319 • PR: https://github.com/flyteorg/flytekit/pull/3429 A heads up for reviewers who worked on flyteorg/flytekit#3394: this PR refactors that path into the strategy abstraction. The public surface (env vars, secret names, fallback order) is unchanged, but
connector.py
now flows through
select_auth
PATAuth.get_bearer_token
get_databricks_token
. Worth opening
connector.py
and
tests/test_databricks_token.py
with that lens. Happy to walk through the design or address review comments. 🙏 cc: @freezing-airport-6809 @glamorous-carpet-83516 @echoing-account-76888
👍 2
f
Cc @echoing-account-76888 @tall-lock-23197 this one too
👀 1
a
Hey @echoing-account-76888 Sorry for the slow update, got stuck in some other internal issues, as you suggested into the PR comment, I’ve split the original Databricks authentication PR into four smaller PRs, all ready for review: 1. PAT auth refactor #3452 2. OAuth M2M #3453 3. Connector OIDC #3454 4. Namespace ServiceAccount OIDC #3455 They are stacked and should be merged in this order. Each PR includes focused tests and documentation.
e
Thank you! cc @faint-pilot-24699 @glamorous-carpet-83516 to have a look when you have time
f
Thanks @adventurous-ability-21671 . Will take a look.
🙏 1
a
Hey @faint-pilot-24699 just checking in if you got chance to take a look into the above PRs?
g
@adventurous-ability-21671 I approved it. are you going to merge 1, and then rebase 2,3,4?
a
Thanks, give me couple of hours will do that.
Hi @glamorous-carpet-83516, I've followed the same approach as the SageMaker stack. PR #3455 now contains all four approved Databricks authentication changes, rebased onto the latest
master
. All local verification and hosted CI checks are green. Please merge only
#3455
. Once merged, I’ll close
#3452
,
#3453
, and
#3454
as included in the final PR.
g
@adventurous-ability-21671 I merged it, thanks for your contributions
a
Thanks @glamorous-carpet-83516 can you please release the version, then I will get our internal release upgraded and test that out to make sure everything is fine.
g
a
Thank you 🙌