https://flyte.org logo
v

Vijay Saravana

06/14/2022, 7:23 PM
Hello guys, Is there a detailed document on Flyte Map tasks. I just see a very superficial page in the website.
k

Ketan (kumare3)

06/14/2022, 7:26 PM
Cc @Dan Rammer (hamersaw) / @Marc Paquette
m

Marc Paquette

06/14/2022, 7:27 PM
Hi @Vijay Saravana. What kind of info are you looking for about map tasks?
v

Vijay Saravana

06/14/2022, 8:02 PM
Hey @Marc Paquette, I am trying to draw a comparison between Spark APIs in Flyte and Flyte Map Tasks. Would like to know the programming model used and it's working in detail.
m

Marc Paquette

06/14/2022, 9:45 PM
@Vijay Saravana Thanks. This page gives more reference details about flytekit.map_task. Do you have specific questions about map_task?
👀 1
v

Vijay Saravana

06/15/2022, 5:21 AM
Thanks @Marc Paquette. Do we have a reduce operation like in pyspark ?
s

Samhita Alla

06/15/2022, 6:28 AM
Hi, @Vijay Saravana! There’s no reduce operation. You can define a task to perform reduce operation on the mapped output. Also, take a look at the blog post on map tasks.
v

Vijay Saravana

06/21/2022, 12:42 AM
Thanks @Samhita Alla, I did read this. Is there any document explaining how parallelism is achieved in Flyte Map tasks ?
s

Samhita Alla

06/21/2022, 4:35 AM
It uses k8s array under the hood by default. Here’s the API reference.
👍 1
v

Vijay Saravana

06/21/2022, 4:41 AM
Thank you so much1
Hello @Samhita Alla, I have a follow-up. Are flyte map tasks only to be called from a workflow or can they be called from a flyte task ? If I call it from a workflow, I see it listed as a map task in Flyte console. Otherwise, when I call it from a task - I see it listed as a python task. But, I am getting the desired results. Does it mean the map task executes properly?
message has been deleted
s

Samhita Alla

06/24/2022, 4:59 PM
Map task should be called from within a workflow. @Marc Paquette is trying to explicitly state this in his current PR: https://github.com/flyteorg/flytesnacks/blob/06a21fad0b17d5f547c4905c4aa144619183329e/cookbook/core/control_flow/map_task.py#L68-L84. Calling a task (or a map task) from within a task isn’t recognized by Flyte as part of the DAG and you wouldn’t be receiving the performance benefits. Indeed, it’s treated as a Python function by Flyte.
3 Views