https://flyte.org logo
Join Slack
Powered by
We're just starting to explore map tasks, and one ...
# flyte-support
e
We're just starting to explore map tasks, and one thing I noticed is that if you use a map task to launch a dynamic step it didn't seem possible to view the full dynamic step as a sub-DAG in the UI. Is there a way to introspect sub-DAGs that are launched via map tasks?
h
Great question. Dynamics aren't actually supported in maptasks. The current maptask only supports Flytes PodPlugin, so using the 
@task
decorator on a python function / 
ContainerTask
/ 
PodTask
/ etc. When you're executing a dynamic task as a maptask it isn't executing the same as how it is as just a regular task where it compiles a sub DAG, etc. It is actually executing everything as a single function call, so tasks inside the dynamic are just python function calls.
e
Oh I see. That makes sense
h
That being said, I am currently working on an 
ArrayNode
which will change how maptasks are executed internally - heres the RFC and I have very minimal working implementation right now.
This means that subtasks will be executed exactly the same as if the task were not in a maptask (ex. supporting intra-task checkpointing, cache serialize, etc). but also means that we can extend this work to s*upport any plugin type or Flyte node type in maptasks,* which is something the current implementation can not support.
Interestingly, this is something that we have thought about internally, but haven't actually any ask from the community. Do you see mapping over other node types (ex. dynamic tasks, subworkflows, etc) as very beneficial?
e
Mapping over other node types, especially dynamic and subworkflows would be very powerful for us
👍 1
This is in the context of back-testing models - we have a workflow to train a model, and we want to run N copies of this workflow for each period we're running the backtest
h
cc @freezing-airport-6809 mapping over other nodes tyes cc @broad-monitor-993 is there a sane way to support this use case right now?
f
@elegant-australia-91422 you can map today simply using dynamic tasks
Is it just the map interface / UX you are looking for
For example @elegant-australia-91422 you can write this
Copy code
@workflow
def sub_wf(i: int):
   ...

@dynamic
def foo(l: List[int]):
  for i in l:
     sub_wf(i=i)

@workflow
def wf(l: List[int]):
   foo(l=l)
This should work right?
e
Looking to take advantage of the improved overhead for map tasks vs dynamic. We’ve seen some perf issues with high fan out dynamic tasks that use a for loop instead of map
👍 1
f
hmm ya, but would love to understand the usecase
also - once dan is done, you could easily contribute this
👍 1
UX is hard
b
is there a sane way to support this use case right now?
Just to follow up on this, there’s currently no way to support running 
map_task
over anything other than a 
@task
… Ketan’s suggestion would be the only way to do this.
h
^^ perfect, didn't know if there was some fancy Flyte trick to do this more efficiently than dynamics 😅
159 Views
Open in Slack
PreviousNext
Powered by