Featured in the latest Flyte v1.3.0 release, the
Flyte-Databricks integration brings together the best of both worlds:
- Flyte supports for type checking, complex inputs/outputs and local execution
- Databricks provides auto-scaling optimization for Spark
Many thanks to
@abundant-hamburger-66584 for showing us how it worked for HBO Max.
https://flyte.org/blog/meet-the-databricks-integration