Channels
#ask-the-community
Title
# ask-the-community
v
varsha Parthasarathy
10/11/2022, 1:53 PMHI team, QQ, for flyte map tasks, we have can use resources to specify the requests and limits, is it also possible to pass in the node pool to run this task on?
Copy code
map_task_partition_outputs = flytekit.map_task(process_map_task_partition, concurrency=300)(
map_task_partition=map_task_partitions).with_overrides(
limits=flytekit.Resources(cpu="4", mem="42Gi"),
requests=flytekit.Resources(cpu="4", mem="42Gi"),
retries=3,
)cc @Samhita Alla
s
Samhita Alla
10/11/2022, 2:20 PM
cc: @Dan Rammer (hamersaw)
y
Yee
10/11/2022, 6:57 PM
hey varsha - just wanted more information, could you elaborate on why please?
there’s no way to set the node pool currently
v
varsha Parthasarathy
10/11/2022, 7:56 PMWe are partitioning our eks node pool across different instance types/team etc
So when we run Map task for a specific project, we would like to provide a node pool to the task to let it know where to run
y
Yee
10/11/2022, 8:12 PM
would it be possible to achieve this with taints and tolerations? @Dan Rammer (hamersaw) do you remember if it’s possible to add tolerations by task type?
adding the concept of node pools explicitly is a bit tricky. flyte aims to be k8s native, but we still try to abstract the internals of k8s away from the user when possible and reasonable. also node pools aren’t a k8s construct right? they’re eks/gke constructs
you can apply labels/annotations to a given project or project-domain https://github.com/flyteorg/flyteidl/blob/master/protos/flyteidl/admin/matchable_resource.proto#L116
but i’m not sure how those tie into node pools
k
katrina
10/11/2022, 8:35 PM
you could use taints and tolerations maybe too
v
varsha Parthasarathy
10/11/2022, 9:00 PMOkay interesting ..
2 Views