https://flyte.org logo
#ask-the-community
Title
# ask-the-community
n

Nick Mostowich

01/18/2023, 8:17 PM
Does the Databricks integration released in 1.3 require usage of Databricks Connect and thus Databricks interactive clusters (vs automated clusters)? Does Flyte introspect into the spark cluster itself, or just rely on the web API?
Or is the requirement Databricks Container services enabled?
k

Kevin Su

01/18/2023, 8:24 PM
flyte doesn’t use databricks connect, it use job api (rest) to submit the job to databricks platform.
n

Nick Mostowich

01/18/2023, 8:24 PM
Very cool - this is potentially huge for us at Altana
k

Kevin Su

01/18/2023, 8:25 PM
yes, it have to enable to use custom image in the spark cluster
awesome. Let me know if you have any question about databricks plugin integration. happy to help.
n

Nick Mostowich

01/18/2023, 8:29 PM
Are databricks dbutils available inside the containers running on databricks spark/on top of dbfs? We have quite a few usages of these inside our jobs and notebooks that I’d like to orchestrate via flyte. I get that these won’t run locally.
k

Kevin Su

01/18/2023, 8:33 PM
spark job is running on ecs cluster created by databricks, you can setup instance profile for you spark job, so that the job can access dbfs data.
e

Evan Sadler

01/18/2023, 8:46 PM
@Nick Mostowich I am currently setting up the databricks plugin for my team. I haven’t checked dbutils yet, but it 100% is available if you can swing using an existing cluster.