Hello, In Airflow there is a mechanism for a dag to wait until another dag has successfully completed (ExternalTaskSensor), then it will start. What is the mechanism in Flyte that achieves this?

You can write your own sensor

Hi @Frank Shen, the User Guide example for creating your own task plugin actually has a sensor example! https://docs.flyte.org/projects/cookbook/en/latest/auto/core/extend_flyte/user_container.html You can customize it for your own use case, e.g. you can configure it with

FlyteRemote

to wait for a task/workflow execution to complete instead of a file in s3 (which is what the example does)

also @Frank Shen some fun stuff is coming soon - infact there is an RFC, this would make your sensors extremely efficient!!! https://hackmd.io/@pingsutw/B1a_Bnfqi

Thanks!

@Niels Bantilan, Could you point me to an example where I can customize a task plugin to configure it with

FlyteRemote

to wait for a task from a different workflow execution to complete? Thanks!

what do you mean - taskplugin with flyte-remote?

@Ketan (kumare3), That’s a very good point. Since both workflows are scheduled jobs (via Launch Plans), one would not know the other’s specific run’s execution id, rather it will know the schedule of the other and therefore the time delta between the two runs. Checking time delta is the mechanism in Airflow’s ExternalTaskSensor. And ExternalTaskSensor is so widely used. I believe Flyte could do the same and fill the gap.

Hmm tbh I find it very confusing

I am investigating a setup that relates to this as well. - Potentially I am just missing some pieces, so I'll just paint a picture.. Imagine Team A collects data with their own launch-plan, workflow and tasks. Then imagine that Team runs their own workflow, but one task depends on the output of a task in A's workflow. Now, if A realized that their input data, or their processing of said data, is bad, we would like to re-run the entirety of "downstream" from the "bad" task. This could be all of A's and all of B's tasks (in Airflow tied to A via a name+time-reference) I think this scenario is one of the major benefits of using a workflow orchestrator. Cross-team cross-workflow is super unruly and will be heavy to manage, if it can't be explicitly orchestrated. Airflow's "rerun everything up/downstream and/or future/past executions" functionality is a huge time saver. It is easy to understand and helps solve all the "I fixed an issue with task x up to/from time y" scenarios. EDIT: Ketan I think your new backfill feature hits at least some of these areas. Maybe you already know how to solve the problem 😉