If our workflow design is poor we welcome recommendations alternate approaches. The reason we are using map_task + dynamic is so we can call a set of tasks not just one from the map_task. However, since map_tasks can only take a single argument, all of my parameters are bundled into one object. When the dynamic task receives the bundled object it calls the sub tasks with just the arguments it cares about.

@dataclass_json
@dataclass
class BundledData:
  arg1: str
  arg2: int

@task
def t1(input: str) -> str:
  print(input)
  return "hello"

@task t2(input: int, msg: str):
  print(input)

@dynamic
def dyn(input: BundledData):
  o = t1(input=input.arg1)
  t2(input=input.arg2, msg=o)

@workflow
def wf(inputs: List[BundledData]):
  map_task(input=inputs)

We should fail in flytekit

One solution might be to have a separate application (outside of flyte) that takes the list of inputs and calls flyte using FlyteRemote to execute a LaunchPlan for each input. However, I’d prefer to have it all happen within Flyte so I could use the FlyteConsole to start processing.

Why not use that?

We should make that clear in the docs cc @Yee

I meant in testing it looks like serial, but when you run on the cluster it will be parallel

Everything in Flyte will parallelize automatically

Map is just a special way of running large arrays

Please suggest edits to docs

Cc @Samhita Alla we should probably explain the model for a workflow with independent data

@Nicholas LoFaso so it's a data flow system, Flyte will make parallel progress as long as data is available and resources are available

E.g, you can write a static for loop in a regular workflow

Can you suggest an edit