Hi all Has there been any work on or interest in scheduling

Question

Hi all Has there been any work on or interest in scheduling workflows based on the presence of input data and the absence of output data I m thinking of something similar to dagster s asset reconciliation where a scheduler is aware of the possibly time partitioned inputs and outputs of each workflow and then it continually schedules a set of workflows as each workflow s input becomes available

Ketan (kumare3) · Answer

hi < Geoff Salmon> firstly welcome to the community Pleasure to meet you

Ketan (kumare3) · Answer

There has been some work that we have talked about for Reactive workflows but not yet done Since all outputs can be looked at independently doing so would be absolutely possible

Ketan (kumare3) · Answer

this was the doc <https docs google com document d 18mviLzJRT UCAllW115LWzZnt1Toe5VoE89TgCeTVzU edit >

Geoff Salmon · Answer

thanks reactive workflows led me to a Flyte OSS Sync Up video from July last year that discusses it Do you know if there s any other public docs or info about reactive workflows Is the latest thinking on it still leveraging Argo Events

Ketan (kumare3) · Answer

not sure if this is exactly what you are looking for

Geoff Salmon · Answer

beat me to it

Ketan (kumare3) · Answer

haha ya

Ketan (kumare3) · Answer

let me know if you want to catch up

Ketan (kumare3) · Answer

in our opinion you can build an API like dagster on top of Flyte just like we did with UnionML

Geoff Salmon · Answer

Yeah it does seem like something that could be built on top Figuring out a nice way for the user to represent the assets and declare their partitioning might be tricky I don t think I know enough about how workflow inputs and outputs are represented currently

Ketan (kumare3) · Answer

hmm i can help

Ketan (kumare3) · Answer

do you have a few minutes to hop on a vc

Ketan (kumare3) · Answer

because i would love to learn what is the usecase

Ketan (kumare3) · Answer

i think it can be done purely on the client side and reactivity can be added later

Geoff Salmon · Answer

sorry I ve got to run now I ll message you when I have more time Thanks for the help and useful pointers

Ketan (kumare3) · Answer

absolutely anytime

Ketan (kumare3) · Answer

always fun to talk about cool things people are thinking about