Hello guys,
Is there a detailed document on Flyte ...
# announcementso
orange-umbrella-99251
06/14/2022, 7:23 PMHello guys,
Is there a detailed document on Flyte Map tasks. I just see a very superficial page in the website.
f
freezing-airport-6809
06/14/2022, 7:26 PM
Cc @hallowed-mouse-14616 / @echoing-queen-96254
e
echoing-queen-96254
06/14/2022, 7:27 PMHi @orange-umbrella-99251. What kind of info are you looking for about map tasks?
o
orange-umbrella-99251
06/14/2022, 8:02 PMHey @echoing-queen-96254, I am trying to draw a comparison between Spark APIs in Flyte and Flyte Map Tasks.
Would like to know the programming model used and it's working in detail.
e
echoing-queen-96254
06/14/2022, 9:45 PM@orange-umbrella-99251 Thanks. This page gives more reference details about flytekit.map_task.
Do you have specific questions about map_task?
👀 1
o
orange-umbrella-99251
06/15/2022, 5:21 AMThanks @echoing-queen-96254. Do we have a reduce operation like in pyspark ?
t
tall-lock-23197
06/15/2022, 6:28 AM
Hi, @orange-umbrella-99251! There’s no reduce operation. You can define a task to perform reduce operation on the mapped output.
Also, take a look at the blog post on map tasks.
o
orange-umbrella-99251
06/21/2022, 12:42 AMThanks @tall-lock-23197, I did read this. Is there any document explaining how parallelism is achieved in Flyte Map tasks ?
t
tall-lock-23197
06/21/2022, 4:35 AM
It uses k8s array under the hood by default. Here’s the API reference.
👍 1
o
orange-umbrella-99251
06/21/2022, 4:41 AMThank you so much1
orange-umbrella-99251
06/24/2022, 4:02 PMHello @tall-lock-23197, I have a follow-up.
Are flyte map tasks only to be called from a workflow or can they be called from a flyte task ? If I call it from a workflow, I see it listed as a map task in Flyte console.
Otherwise, when I call it from a task - I see it listed as a python task.
But, I am getting the desired results. Does it mean the map task executes properly?
orange-umbrella-99251
06/24/2022, 4:04 PMmessage has been deleted
t
tall-lock-23197
06/24/2022, 4:59 PM
Map task should be called from within a workflow. @echoing-queen-96254 is trying to explicitly state this in his current PR: https://github.com/flyteorg/flytesnacks/blob/06a21fad0b17d5f547c4905c4aa144619183329e/cookbook/core/control_flow/map_task.py#L68-L84.
Calling a task (or a map task) from within a task isn’t recognized by Flyte as part of the DAG and you wouldn’t be receiving the performance benefits. Indeed, it’s treated as a Python function by Flyte.
165 Views