Does the Databricks integration released in 1.3 require usage of Databricks Connect and thus Databricks interactive clusters (vs automated clusters)? Does Flyte introspect into the spark cluster itself, or just rely on the web API?

flyte doesn’t use databricks connect, it use job api (rest) to submit the job to databricks platform.

Very cool - this is potentially huge for us at Altana

yes, it have to enable to use custom image in the spark cluster

Are databricks dbutils available inside the containers running on databricks spark/on top of dbfs? We have quite a few usages of these inside our jobs and notebooks that I’d like to orchestrate via flyte. I get that these won’t run locally.

spark job is running on ecs cluster created by databricks, you can setup instance profile for you spark job, so that the job can access dbfs data.

@Nick Mostowich I am currently setting up the databricks plugin for my team. I haven’t checked dbutils yet, but it 100% is available if you can swing using an existing cluster.