https://flyte.org logo
Join Slack
Powered by
Hi all. Has there been any work on, or interest in...
# announcements
n
Hi all. Has there been any work on, or interest in, scheduling workflows based on the presence of input data and the absence of output data? I'm thinking of something similar to dagster's "asset reconciliation" where a scheduler is aware of the (possibly time-partitioned) inputs and outputs of each workflow and then it continually schedules a set of workflows as each workflow's input becomes available.
f
hi @nutritious-london-39005 firstly welcome to the community. Pleasure to meet you
thx 1
There has been some work that we have talked about for Reactive workflows, but not yet done. Since all outputs can be looked at independently, doing so would be absolutely possible.
n
thanks. "reactive workflows" led me to a Flyte OSS Sync Up video from July last year that discusses it. Do you know if there's any other public docs or info about reactive workflows? Is the latest thinking on it still leveraging Argo Events?
f
not sure if this is exactly what you are looking for
n
beat me to it!
f
haha ya
let me know if you want to catch up
in our opinion you can build an API like dagster on top of Flyte, just like we did with UnionML
n
Yeah, it does seem like something that could be built on top. Figuring out a nice way for the user to represent the "assets" and declare their partitioning might be tricky. I don't think I know enough about how workflow inputs and outputs are represented currently.
f
hmm i can help
do you have a few minutes to hop on a vc
because i would love to learn, what is the usecase
i think it can be done purely on the client side and reactivity can be added later
n
sorry, I've got to run now. I'll message you when I have more time. Thanks for the help and useful pointers.
f
absolutely - anytime
always fun to talk about cool things people are thinking about
👍 2
168 Views
Open in Slack
PreviousNext
Powered by