ancient-wolf-19325
09/06/2024, 6:54 PMfrom flytekit import dynamic, task, workflow, map_task, PythonFunctionTask
@task
def t1(foo: list[int]) -> None:
for i in foo:
print(i)
@task
def t2(bar: list[int]) -> None:
for i in bar:
print(i)
@dynamic
def d1(task_function_to_map: PythonFunctionTask, l: dict[str, list[int]]) -> None:
# Do common stuff
map_task(task_function_to_map)(**l)
# Do more common stuff
@workflow
def main_wf() -> None:
d1(
task_function_to_map=t1,
l={"foo": [1, 2, 3, 4, 5, 6, 7, 8, 9]}
)
d1(
task_function_to_map=t2,
l={"bar": [1, 2, 3, 4, 5, 6, 7, 8, 9]}
)
This doesn't work as the arraynode fails to bind the variable in map task. Is there an elegant way to do this ?
thx in advance
# Answer
Currently, map_task in Flyte does not support dynamic tasks or workflows as inputs. This limitation is acknowledged in various discussions and issues within the Flyte community. According to this discussion, map_task only supports Flyte's PodPlugin tasks, such as Python tasks, container tasks, and pod tasks.
A potential workaround is to use dynamic tasks within a static workflow, but this does not fully address the need for mapping over different task functions dynamically. The Flyte team is working on extending support for mapping over different Flyte node types, including dynamic and subworkflows, through the introduction of ArrayNode (source). This feature is still under development and not yet available.
Would you like more details on the current limitations or the upcoming ArrayNode feature?
Sources:
- Flyte Discussion on Dynamic Workflows
- Flyte Issue on MapTasks
Slack Link
https://flyte-org.slack.com/archives/CP2HDHKE1/p1725648868.887149 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.