Thread
#announcements
    m

    Miggy

    5 months ago
    There’s a new use case that came up and we’re trying to figure how best to do this. I was wondering if I could some help with resources/ideas. So we have a bunch of tasks that are already registered and at runtime we want to be able to construct the graph. One thought was to basically create the workflow graph spec and submit a request to flyteadmin similar to how flytekit does. Then the newly created workflow can be launched. In short we want to be able to generate a workflow without needing to write python code for it. Just specify the proto spec. Thoughts? @katrina @Ketan (kumare3) @Yee cc: @Rupsha Chaudhuri @Danielle Curammeng
    Yee

    Yee

    5 months ago
    can’t you fetch_task?
    and then imperative workflow it out?
    does that work? (i mean conceptually… the flytekit code might have some bugs since this is pretty corner-case-y for now)
    but that is still code.
    m

    Miggy

    5 months ago
    So it would kind of work but then you still have to generate the code for it at runtime. The idea is there is a UI that users can then move around to build their workflow and then the service would turn this UI to a graph. @Danielle Curammeng feel free to add anything that I may have missed
    Yee

    Yee

    5 months ago
    you can yeah
    something has to handle the dag structure creation
    if you don’t want flytekit to do it, then the JS would have to
    are you just making a graphical workflow creator?
    Danielle Curammeng

    Danielle Curammeng

    5 months ago
    yes basically
    Yee

    Yee

    5 months ago
    coool
    can you contribute this back to open source 🙂
    m

    Miggy

    5 months ago
    Yeah it’ll have to be done in JS in some way. I don’t see why the core piece of it can’t be open sourced.
    Is there any specific docs or what the payload would look like for @Danielle Curammeng to take a look at?
    Danielle Curammeng

    Danielle Curammeng

    5 months ago
    just to clarify, so we need to handle dag structure creation, but we’d be able to do it in such a way that the actual node dependencies (ordering, parallelism, etc.) are handled by flyte during the workflow execution?
    Yee

    Yee

    5 months ago
    if the dag is created correctly, then yeah the flyte execution backend will handle everything else
    but the creation of the dag structure is not trivial
    in terms of documentation, i think the best thing to look at is the protobuf that admin expects
    Ketan (kumare3)

    Ketan (kumare3)

    5 months ago
    @Miggy you can actually use imperative programming model instead of going to the proto
    m

    Miggy

    5 months ago
    Yeah but I think the goal here is ideally go to proto. The imperative model there still has to be a way of generating the dag and inputs and outputs dynamically so going straight to proto from UI would I think be a better solution
    Ketan (kumare3)

    Ketan (kumare3)

    5 months ago
    ya would we awesome
    we will add it to flyteconsole
    @Miggy can @Danielle Curammeng write it in flyteconsole itself?
    they can probably fork it
    and they can get help from @Jason Porter 🙂
    Danielle Curammeng

    Danielle Curammeng

    5 months ago
    Hi all, sorry for digging up an old thread, but I wanted to circle back to this idea. For my current use case, we have our own definitions of nodes/workflows as well, so I’m thinking a conversion tool would make more sense (than, say, working directly in flyteconsole). However I was wondering if I could still keep some of you in the loop for support/tips in creating the protos. Also, any chance there are examples anywhere? (does flytekit or flyteconsole interact with flyteadmin this way?)
    Ketan (kumare3)

    Ketan (kumare3)

    5 months ago
    @Danielle Curammeng lets make a channel, where we can collaborate. We support this
    Danielle Curammeng

    Danielle Curammeng

    5 months ago
    Just created #workflow-building-ui-proj!