https://flyte.org logo
Powered by
Title
r

Rahul Mehta

04/14/2023, 6:22 PM
We're just starting to explore map tasks, and one thing I noticed is that if you use a map task to launch a dynamic step it didn't seem possible to view the full dynamic step as a sub-DAG in the UI. Is there a way to introspect sub-DAGs that are launched via map tasks?
d

Dan Rammer (hamersaw)

04/14/2023, 7:27 PM
Great question. Dynamics aren't actually supported in maptasks. The current maptask only supports Flytes PodPlugin, so using the 
@task
decorator on a python function / 
ContainerTask
/ 
PodTask
/ etc. When you're executing a dynamic task as a maptask it isn't executing the same as how it is as just a regular task where it compiles a sub DAG, etc. It is actually executing everything as a single function call, so tasks inside the dynamic are just python function calls.
r

Rahul Mehta

04/14/2023, 7:27 PM
Oh I see. That makes sense
d

Dan Rammer (hamersaw)

04/14/2023, 7:28 PM
That being said, I am currently working on an 
ArrayNode
which will change how maptasks are executed internally - heres the RFC and I have very minimal working implementation right now.
This means that subtasks will be executed exactly the same as if the task were not in a maptask (ex. supporting intra-task checkpointing, cache serialize, etc). but also means that we can extend this work to s*upport any plugin type or Flyte node type in maptasks,* which is something the current implementation can not support.
Interestingly, this is something that we have thought about internally, but haven't actually any ask from the community. Do you see mapping over other node types (ex. dynamic tasks, subworkflows, etc) as very beneficial?
r

Rahul Mehta

04/14/2023, 7:32 PM
Mapping over other node types, especially dynamic and subworkflows would be very powerful for us
This is in the context of back-testing models - we have a workflow to train a model, and we want to run N copies of this workflow for each period we're running the backtest
d

Dan Rammer (hamersaw)

04/14/2023, 7:37 PM
cc @Ketan (kumare3) mapping over other nodes tyes cc @Niels Bantilan is there a sane way to support this use case right now?
k

Ketan (kumare3)

04/15/2023, 5:11 AM
@Rahul Mehta you can map today simply using dynamic tasks
Is it just the map interface / UX you are looking for
For example @Rahul Mehta you can write this 
@workflow
def sub_wf(i: int):
   ...

@dynamic
def foo(l: List[int]):
  for i in l:
     sub_wf(i=i)

@workflow
def wf(l: List[int]):
   foo(l=l)
This should work right?
r

Rahul Mehta

04/16/2023, 12:12 AM
Looking to take advantage of the improved overhead for map tasks vs dynamic. We’ve seen some perf issues with high fan out dynamic tasks that use a for loop instead of map
k

Ketan (kumare3)

04/16/2023, 12:50 AM
hmm ya, but would love to understand the usecase
also - once dan is done, you could easily contribute this
UX is hard
n

Niels Bantilan

04/17/2023, 6:28 PM
is there a sane way to support this use case right now?
Just to follow up on this, there’s currently no way to support running 
map_task
over anything other than a 
@task
… Ketan’s suggestion would be the only way to do this.
d

Dan Rammer (hamersaw)

04/17/2023, 6:42 PM
^^ perfect, didn't know if there was some fancy Flyte trick to do this more efficiently than dynamics 😅
#ask-the-communityJoin Slack
Powered by