We re just starting to explore map tasks and one thing I not Flyte #ask-the-community

We're just starting to explore map tasks, and one ...

Rahul Mehta

04/14/2023, 6:22 PM

We're just starting to explore map tasks, and one thing I noticed is that if you use a map task to launch a dynamic step it didn't seem possible to view the full dynamic step as a sub-DAG in the UI. Is there a way to introspect sub-DAGs that are launched via map tasks?

Dan Rammer (hamersaw)

04/14/2023, 7:27 PM

Great question. Dynamics aren't actually supported in maptasks. The current maptask only supports Flytes PodPlugin, so using the

@task

decorator on a python function /

ContainerTask

PodTask

/ etc. When you're executing a dynamic task as a maptask it isn't executing the same as how it is as just a regular task where it compiles a sub DAG, etc. It is actually executing everything as a single function call, so tasks inside the dynamic are just python function calls.

Rahul Mehta

04/14/2023, 7:27 PM

Oh I see. That makes sense

Dan Rammer (hamersaw)

04/14/2023, 7:28 PM

That being said, I am currently working on an

ArrayNode

which will change how maptasks are executed internally - heres the RFC and I have very minimal working implementation right now.

Dan Rammer (hamersaw)

04/14/2023, 7:29 PM

This means that subtasks will be executed exactly the same as if the task were not in a maptask (ex. supporting intra-task checkpointing, cache serialize, etc). but also means that we can extend this work to s*upport any plugin type or Flyte node type in maptasks,* which is something the current implementation can not support.

Dan Rammer (hamersaw)

04/14/2023, 7:31 PM

Interestingly, this is something that we have thought about internally, but haven't actually any ask from the community. Do you see mapping over other node types (ex. dynamic tasks, subworkflows, etc) as very beneficial?

Rahul Mehta

04/14/2023, 7:32 PM

Mapping over other node types, especially dynamic and subworkflows would be very powerful for us

Rahul Mehta

04/14/2023, 7:32 PM

This is in the context of back-testing models - we have a workflow to train a model, and we want to run N copies of this workflow for each period we're running the backtest

Dan Rammer (hamersaw)

04/14/2023, 7:37 PM

cc @Ketan (kumare3) mapping over other nodes tyes cc @Niels Bantilan is there a sane way to support this use case right now?

Ketan (kumare3)

04/15/2023, 5:11 AM

@Rahul Mehta you can map today simply using dynamic tasks

Ketan (kumare3)

04/15/2023, 5:12 AM

Is it just the map interface / UX you are looking for

Ketan (kumare3)

04/15/2023, 11:44 PM

For example @Rahul Mehta you can write this

Copy code

@workflow
def sub_wf(i: int):
   ...

@dynamic
def foo(l: List[int]):
  for i in l:
     sub_wf(i=i)

@workflow
def wf(l: List[int]):
   foo(l=l)

Ketan (kumare3)

04/15/2023, 11:44 PM

This should work right?

Rahul Mehta

04/16/2023, 12:12 AM

Looking to take advantage of the improved overhead for map tasks vs dynamic. We’ve seen some perf issues with high fan out dynamic tasks that use a for loop instead of map

Ketan (kumare3)

04/16/2023, 12:50 AM

hmm ya, but would love to understand the usecase

Ketan (kumare3)

04/16/2023, 12:50 AM

also - once dan is done, you could easily contribute this

Ketan (kumare3)

04/16/2023, 12:50 AM

UX is hard

Niels Bantilan

04/17/2023, 6:28 PM

is there a sane way to support this use case right now?

Just to follow up on this, there’s currently no way to support running

map_task

over anything other than a

@task

… Ketan’s suggestion would be the only way to do this.

Dan Rammer (hamersaw)

04/17/2023, 6:42 PM

^^ perfect, didn't know if there was some fancy Flyte trick to do this more efficiently than dynamics 😅

156 Views

Open in Slack

Previous Next