Is there a way to use

DatabricksRunNowOperator

and other databricks operators of airflow, in flyte: https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/operators/run_now.html?

cc: @glamorous-carpet-83516

we haven’t tested databricks operator, but please give it a try. you can directly use that operator in the flyte workflow. like https://docs.flyte.org/en/latest/user_guide/development_lifecycle/migrating_from_airflow_to_flyte.html#id1

Will give it a shot. But is there a way to “trigger” databricks jobs (and wait for result) natively from flyte ? I see there is Databricks Agent and some spark plugins but not enough examples to understand how to do it exactly.

have you seen this example? https://docs.flyte.org/en/latest/flytesnacks/examples/databricks_agent/databricks_agent_example_usage.html

yes, in this example the code given in the function is running. I already have a databricks job defined. I just want to trigger it

you mean how to run the example?

pyflyte run --remote my_databricks_job wf

No. Like in databricks you can have pre-defined jobs right. (Code resides in databricks). Now I want to run it as a task in my workflow. I want my workflow to fail if databricks job fails and terminate if I terminate the workflow. This exact behaviour is implemented in

DatabricksRunNowOperator

of airflow. I want similar thing in flyte

ah, sorry. we don’t have that kind operator natively in flyte. you could add a custom agent to do trigger your predefined jobs in databricks.