Thread
#announcements
    g

    Geoff Salmon

    1 month ago
    Hi all. Has there been any work on, or interest in, scheduling workflows based on the presence of input data and the absence of output data? I'm thinking of something similar to dagster's "asset reconciliation" where a scheduler is aware of the (possibly time-partitioned) inputs and outputs of each workflow and then it continually schedules a set of workflows as each workflow's input becomes available.
    Ketan (kumare3)

    Ketan (kumare3)

    1 month ago
    hi @Geoff Salmon firstly welcome to the community. Pleasure to meet you
    There has been some work that we have talked about for Reactive workflows, but not yet done. Since all outputs can be looked at independently, doing so would be absolutely possible.
    g

    Geoff Salmon

    1 month ago
    thanks. "reactive workflows" led me to a Flyte OSS Sync Up video from July last year that discusses it. Do you know if there's any other public docs or info about reactive workflows? Is the latest thinking on it still leveraging Argo Events?
    Ketan (kumare3)

    Ketan (kumare3)

    1 month ago
    not sure if this is exactly what you are looking for
    g

    Geoff Salmon

    1 month ago
    beat me to it!
    Ketan (kumare3)

    Ketan (kumare3)

    1 month ago
    haha ya
    let me know if you want to catch up
    in our opinion you can build an API like dagster on top of Flyte, just like we did with UnionML
    g

    Geoff Salmon

    1 month ago
    Yeah, it does seem like something that could be built on top. Figuring out a nice way for the user to represent the "assets" and declare their partitioning might be tricky. I don't think I know enough about how workflow inputs and outputs are represented currently.
    Ketan (kumare3)

    Ketan (kumare3)

    1 month ago
    hmm i can help
    do you have a few minutes to hop on a vc
    because i would love to learn, what is the usecase
    i think it can be done purely on the client side and reactivity can be added later
    g

    Geoff Salmon

    1 month ago
    sorry, I've got to run now. I'll message you when I have more time. Thanks for the help and useful pointers.
    Ketan (kumare3)

    Ketan (kumare3)

    1 month ago
    absolutely - anytime
    always fun to talk about cool things people are thinking about