Hi Team, My team at Ripple uses Databricks as main ML Platform. We have been trying to figure out ways to parallelize some of our compute efforts. I am wondering if Databricks Flyte integration would be helpful in this case?
I am trying to understand what are the benefits of Flyte/Databricks integration? Our Databricks is deployed on AWS.
01/23/2024, 8:10 PM
hey @Shahwar Saleem yes, it can parallelize, manage clusters, you can reuse clusters, run jobs, allow you to cache results and also modify code on demand and run them
it basically makes iterating on Databricks much faster, with imagespec, fast registration and (we recommend starting with Agents)
you can also add jobs that are non databricks that are using k8s containers
Happy to connect and walk over things.
There are many folks that are using databricks and Flyte together
01/23/2024, 8:12 PM
This is really awesome, this would be my second POC with Flyte. (Good to see you again)
Is there any documentation that I can educate myself with before I start meeting 1:1?
01/23/2024, 8:13 PM
haha ohh ya, thank you for joining us again 😄 pleasure to work