Hey all! We are missing a flyte-native mechanism for executing our scheduled workflows based on delayed external upstream dependencies. This mechanism should: On a schedule kick-in: • check if the dependencies (S3 file or S3 prefix) are in place or not; • if the dependencies are ready and the workflow is not running yet - run the workflow; • if the dependencies are ready and there is a workflow running already - do nothing; • if the dependencies are not ready - retry N times with an exponential delay and otherwise log message and do graceful exit (terminate the workflow);

@David Espejo (he/him), many of our users have been requesting reactive scheduling, so I think it would be worthwhile to discuss it in the upcoming contributors' meeting. There is also a related issue that you can refer to: https://github.com/flyteorg/flyte/discussions/3754

@Samhita Alla would this also cover points 1 and 4: • check if the dependencies (S3 file or S3 prefix) are in place or not; • if the dependencies are not ready - retry N times with an exponential delay and otherwise log message and do graceful exit (terminate the workflow); Or are there some other ways of doing this?

@Samhita Alla this feature seems like a concurrencyPolicy implementation for LaunchPlans which we need as well But what I wanted to emphasize is the need of a sensor approach for upstream dependencies

Thanks everyone here for the input. I just added this to the agenda @Mücahit @Maria Slanova you're all welcome to join next week's contributors meetup to discuss this request directly with the maintainers, steering committee and other contributors. There's also an RFC process in place that typically starts as a Discussion

Thanks David! Will join! Just to double-check - there’s no meeting this week?