Channels
#announcements
Title
v
Vijay Saravana
06/14/2022, 7:23 PMHello guys,
Is there a detailed document on Flyte Map tasks. I just see a very superficial page in the website.
k
Ketan (kumare3)
06/14/2022, 7:26 PM
Cc @Dan Rammer (hamersaw) / @Marc Paquette
m
Marc Paquette
06/14/2022, 7:27 PMHi @Vijay Saravana. What kind of info are you looking for about map tasks?
v
Vijay Saravana
06/14/2022, 8:02 PMHey @Marc Paquette, I am trying to draw a comparison between Spark APIs in Flyte and Flyte Map Tasks.
Would like to know the programming model used and it's working in detail.
m
Marc Paquette
06/14/2022, 9:45 PM@Vijay Saravana Thanks. This page gives more reference details about flytekit.map_task.
Do you have specific questions about map_task?
👀 1
v
Vijay Saravana
06/15/2022, 5:21 AMThanks @Marc Paquette. Do we have a reduce operation like in pyspark ?
s
Samhita Alla
06/15/2022, 6:28 AM
Hi, @Vijay Saravana! There’s no reduce operation. You can define a task to perform reduce operation on the mapped output.
Also, take a look at the blog post on map tasks.
v
Vijay Saravana
06/21/2022, 12:42 AMThanks @Samhita Alla, I did read this. Is there any document explaining how parallelism is achieved in Flyte Map tasks ?
s
Samhita Alla
06/21/2022, 4:35 AM
It uses k8s array under the hood by default. Here’s the API reference.
👍 1
v
Vijay Saravana
06/21/2022, 4:41 AMThank you so much1
Hello @Samhita Alla, I have a follow-up.
Are flyte map tasks only to be called from a workflow or can they be called from a flyte task ? If I call it from a workflow, I see it listed as a map task in Flyte console.
Otherwise, when I call it from a task - I see it listed as a python task.
But, I am getting the desired results. Does it mean the map task executes properly?
message has been deleted
s
Samhita Alla
06/24/2022, 4:59 PM
Map task should be called from within a workflow. @Marc Paquette is trying to explicitly state this in his current PR: https://github.com/flyteorg/flytesnacks/blob/06a21fad0b17d5f547c4905c4aa144619183329e/cookbook/core/control_flow/map_task.py#L68-L84.
Calling a task (or a map task) from within a task isn’t recognized by Flyte as part of the DAG and you wouldn’t be receiving the performance benefits. Indeed, it’s treated as a Python function by Flyte.
3 Views