https://flyte.org logo
#ask-the-community
Title
# ask-the-community
t

Tim Hebbeler

04/26/2023, 5:54 PM
Hi everyone, Is it possible to run part of a worklow or a list of tasks in a single pod? I would like to use the caching mechanism for tasks but reduce the cost of starting a pod for each task. I have a task that i want to call up thousands of times, but it doesn't require a lot of computing time
d

David Espejo (he/him)

04/26/2023, 9:34 PM
hi @Tim Hebbeler have you checked Map Tasks? Your use case sounds like a fit https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/map_task.html
t

Tim Hebbeler

04/26/2023, 10:10 PM
But from what i read in the K8s Array plugin a pod is started for each instance in the array.
k

Kevin Su

04/26/2023, 10:18 PM
you want to reuse the same pod to run the other tasks, right?Flyte doesn’t support it yet.
t

Tim Hebbeler

04/26/2023, 10:24 PM
yes, exactly
k

Kevin Su

04/26/2023, 10:30 PM
cc @Ketan (kumare3)
t

Tim Hebbeler

04/26/2023, 10:45 PM
thank you both very much for the replies. Do you think it is enough to just adapt the k8s array plugin or are extensive changes necessary?
k

Ketan (kumare3)

04/26/2023, 11:40 PM
No so we have been thinking of this
@Dan Rammer (hamersaw) is almost done completely reengineering array/map tasks, this will give us the capability of reusing pods
We would Love to work with you if you can contribute
Ohh you want to call a 1000 times, I also think this is a good case for flytekit plugins
Backend flytekit plugin I mean
28 Views