https://flyte.org logo
#ask-the-community
Title
# ask-the-community
v

varsha Parthasarathy

10/11/2022, 1:53 PM
HI team, QQ, for flyte map tasks, we have can use resources to specify the requests and limits, is it also possible to pass in the node pool to run this task on?
Copy code
map_task_partition_outputs = flytekit.map_task(process_map_task_partition, concurrency=300)(
        map_task_partition=map_task_partitions).with_overrides(
            limits=flytekit.Resources(cpu="4", mem="42Gi"),
            requests=flytekit.Resources(cpu="4", mem="42Gi"),
            retries=3,
    )
Example,== ^^^ is one of our map task function call.
cc @Samhita Alla
s

Samhita Alla

10/11/2022, 2:20 PM
cc: @Dan Rammer (hamersaw)
y

Yee

10/11/2022, 6:57 PM
hey varsha - just wanted more information, could you elaborate on why please?
there’s no way to set the node pool currently
v

varsha Parthasarathy

10/11/2022, 7:56 PM
We are partitioning our eks node pool across different instance types/team etc So when we run Map task for a specific project, we would like to provide a node pool to the task to let it know where to run
y

Yee

10/11/2022, 8:12 PM
would it be possible to achieve this with taints and tolerations? @Dan Rammer (hamersaw) do you remember if it’s possible to add tolerations by task type?
adding the concept of node pools explicitly is a bit tricky. flyte aims to be k8s native, but we still try to abstract the internals of k8s away from the user when possible and reasonable. also node pools aren’t a k8s construct right? they’re eks/gke constructs
you can apply labels/annotations to a given project or project-domain https://github.com/flyteorg/flyteidl/blob/master/protos/flyteidl/admin/matchable_resource.proto#L116
but i’m not sure how those tie into node pools
k

katrina

10/11/2022, 8:35 PM
you could use taints and tolerations maybe too
v

varsha Parthasarathy

10/11/2022, 9:00 PM
Okay interesting ..
2 Views