Hey everyone! 👋 I've been working on adding Databricks Serverless Compute and Unity Catalog support to the Databricks connector. We collaborated with the Databricks team on this, and it's now tested and deployed in our environment. What it enables: • Faster job startup (15 min → under 1 min with serverless) • Fine-grained, multi-tenant authentication via Unity Catalog Issues: • <http://• https://github.com/flyteorg/flyte/issues/6911|[Core feature] Add Databricks Serverless Compute Support to Databricks Connector> • <http://• https://github.com/flyteorg/flyte/issues/6912|[Core feature] Fine-Grained Token Support for Multi-Tenant Databricks Authentication> PRs: • Add Databricks Serverless Compute Support • Add serverless-compatible entrypoint for Databricks • Add multi-tenant Databricks token support via cross-namespace K8s secrets • Add RBAC support for cross-namespace secret reading Happy to provide any additional context for review. Feedback welcome. cc: @freezing-airport-6809 /@victorious-zoo-70738 /@mysterious-country-97979 /@helpful-mechanic-41804

cc @echoing-account-76888 / @glamorous-carpet-83516 can you help with the PRs?

Sure! I'll help on this 🙌

Thanks, @echoing-account-76888 let me know if you want any other details from my end.

@adventurous-ability-21671 thanks for opening PRs, could you also fix the lint errors in this PR https://github.com/flyteorg/flytekit/pull/3394

Thanks @glamorous-carpet-83516 for reviewing the PRs I have done following: • 3394 - Fixed the lint issue and pushed the changes. • 6919 - Ran the

make helm

which generated HELM Docs and updated the

values.yaml

Let me now, if changes looks good now.

https://github.com/flyteorg/flytekit/pull/3394 still has lint errors. you could run

make lint

to test it locally

@adventurous-ability-21671 Thanks for the PR! https://github.com/flyteorg/flytekit/pull/3392 overall LGTM, just left 2 small comments. Could you also fix the lint errors in this PR?

Hey @glamorous-carpet-83516, Thanks for re running the pipeline, updated PR #3394 with the lint fix. The issue was a missing

pydoclint-errors-baseline.txt

this file tracks pre-existing docstring violations repo-wide (per #3077) and was empty on our branch. Restored it from `master`; pydoclint auto-shrunk it by 4 lines since our code actually fixed the DatabricksV2 docstring. No new violations introduced.

make lint

passes cleanly now.

$ make lint                 
mypy flytekit/core
Success: no issues found in 52 source files
mypy flytekit/types
Success: no issues found in 23 source files
mypy --allow-empty-bodies --disable-error-code="annotation-unchecked" tests/flytekit/unit/core
Success: no issues found in 102 source files
pre-commit run --all-files
ruff.....................................................................Passed
ruff-format..............................................................Passed
check yaml...............................................................Passed
fix end of files.........................................................Passed
trim trailing whitespace.................................................Passed
shellcheck...............................................................Passed
Check for exposed PDB statements.........................................Passed
codespell................................................................Passed
pydoclint................................................................Passed

Hey man sorry but I will ask them it lgtm they should merge. Cc @echoing-account-76888 / @glamorous-carpet-83516 please help him merge

Hi @adventurous-ability-21671, Sorry for the delay! I approved and merged all your PRs except https://github.com/flyteorg/flytekit/pull/3394, as there's conflict after merging your #3392. Could you please solve the conflict? Thank you!

Hey @echoing-account-76888 Thanks for merging the PRs, I have resolved the conflicts as well as I can see now CI is passing, can you please check 3349 again?

@adventurous-ability-21671 merged! Thank you!

Thanks @echoing-account-76888 for merging it 🙏