Channels
#announcements
Title
m
Miggy
03/29/2022, 10:00 PMThere’s a new use case that came up and we’re trying to figure how best to do this. I was wondering if I could some help with resources/ideas. So we have a bunch of tasks that are already registered and at runtime we want to be able to construct the graph. One thought was to basically create the workflow graph spec and submit a request to flyteadmin similar to how flytekit does. Then the newly created workflow can be launched.
In short we want to be able to generate a workflow without needing to write python code for it. Just specify the proto spec. Thoughts? @katrina @Ketan (kumare3) @Yee
cc: @Rupsha Chaudhuri @Danielle Curammeng
y
Yee
03/29/2022, 10:02 PM
can’t you fetch_task?
and then imperative workflow it out?
does that work? (i mean conceptually… the flytekit code might have some bugs since this is pretty corner-case-y for now)
but that is still code.
m
Miggy
03/29/2022, 10:04 PMSo it would kind of work but then you still have to generate the code for it at runtime. The idea is there is a UI that users can then move around to build their workflow and then the service would turn this UI to a graph. @Danielle Curammeng feel free to add anything that I may have missed
👍 1
y
Yee
03/29/2022, 10:05 PM
you can yeah
something has to handle the dag structure creation
if you don’t want flytekit to do it, then the JS would have to
are you just making a graphical workflow creator?
d
Danielle Curammeng
03/29/2022, 10:06 PMyes basically
y
Yee
03/29/2022, 10:06 PM
coool
can you contribute this back to open source 🙂
m
Miggy
03/29/2022, 10:20 PMYeah it’ll have to be done in JS in some way. I don’t see why the core piece of it can’t be open sourced.
Is there any specific docs or what the payload would look like for @Danielle Curammeng to take a look at?
d
Danielle Curammeng
03/29/2022, 10:26 PMjust to clarify, so we need to handle dag structure creation, but we’d be able to do it in such a way that the actual node dependencies (ordering, parallelism, etc.) are handled by flyte during the workflow execution?
y
Yee
03/29/2022, 10:33 PM
if the dag is created correctly, then yeah the flyte execution backend will handle everything else
but the creation of the dag structure is not trivial
in terms of documentation, i think the best thing to look at is the protobuf that admin expects
k
Ketan (kumare3)
03/29/2022, 10:42 PM
@Miggy you can actually use imperative programming model instead of going to the proto
m
Miggy
03/29/2022, 11:18 PMYeah but I think the goal here is ideally go to proto. The imperative model there still has to be a way of generating the dag and inputs and outputs dynamically so going straight to proto from UI would I think be a better solution
k
Ketan (kumare3)
03/30/2022, 12:06 AM
ya would we awesome
we will add it to flyteconsole
@Miggy can @Danielle Curammeng write it in flyteconsole itself?
they can probably fork it
and they can get help from @Jason Porter 🙂
d
Danielle Curammeng
04/26/2022, 7:06 PMHi all, sorry for digging up an old thread, but I wanted to circle back to this idea. For my current use case, we have our own definitions of nodes/workflows as well, so I’m thinking a conversion tool would make more sense (than, say, working directly in flyteconsole).
However I was wondering if I could still keep some of you in the loop for support/tips in creating the protos. Also, any chance there are examples anywhere? (does flytekit or flyteconsole interact with flyteadmin this way?)
k
Ketan (kumare3)
04/26/2022, 7:40 PM
@Danielle Curammeng lets make a channel, where we can collaborate. We support this
d
Danielle Curammeng
04/26/2022, 8:20 PMJust created #workflow-building-ui-proj!
3 Views