Hi everyone,
Is it possible to run part of a workl...
# flyte-supportb
breezy-receptionist-39475
04/26/2023, 5:54 PMHi everyone,
Is it possible to run part of a worklow or a list of tasks in a single pod? I would like to use the caching mechanism for tasks but reduce the cost of starting a pod for each task. I have a task that i want to call up thousands of times, but it doesn't require a lot of computing time
a
average-finland-92144
04/26/2023, 9:34 PMhi @breezy-receptionist-39475
have you checked Map Tasks? Your use case sounds like a fit
https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/map_task.html
b
breezy-receptionist-39475
04/26/2023, 10:10 PMBut from what i read in the K8s Array plugin a pod is started for each instance in the array.
g
glamorous-carpet-83516
04/26/2023, 10:18 PMyou want to reuse the same pod to run the other tasks, right?Flyte doesn’t support it yet.
b
breezy-receptionist-39475
04/26/2023, 10:24 PMyes, exactly
g
glamorous-carpet-83516
04/26/2023, 10:30 PMcc @freezing-airport-6809
b
breezy-receptionist-39475
04/26/2023, 10:45 PMthank you both very much for the replies. Do you think it is enough to just adapt the k8s array plugin or are extensive changes necessary?
f
freezing-airport-6809
04/26/2023, 11:40 PM
No so we have been thinking of this
freezing-airport-6809
04/26/2023, 11:41 PM
@hallowed-mouse-14616 is almost done completely reengineering array/map tasks, this will give us the capability of reusing pods
freezing-airport-6809
04/26/2023, 11:41 PM
We would
Love to work with you if you can contribute
freezing-airport-6809
04/26/2023, 11:42 PM
Ohh you want to call a 1000 times, I also think this is a good case for flytekit plugins
freezing-airport-6809
04/26/2023, 11:42 PM
Backend flytekit plugin I mean
183 Views