https://flyte.org
Join Slack
HI team, QQ, for flyte map tasks, we have can use resources to specify the requests and limits, is i...
# flyte-support
s
HI team, QQ, for flyte map tasks, we have can use resources to specify the requests and limits, is it also possible to pass in the node pool to run this task on?
Copy code
map_task_partition_outputs = flytekit.map_task(process_map_task_partition, concurrency=300)(
        map_task_partition=map_task_partitions).with_overrides(
            limits=flytekit.Resources(cpu="4", mem="42Gi"),
            requests=flytekit.Resources(cpu="4", mem="42Gi"),
            retries=3,
    )
Example,== ^^^ is one of our map task function call.
cc @tall-lock-23197
t
cc: @hallowed-mouse-14616
t
hey varsha - just wanted more information, could you elaborate on why please?
there’s no way to set the node pool currently
s
We are partitioning our eks node pool across different instance types/team etc So when we run Map task for a specific project, we would like to provide a node pool to the task to let it know where to run
t
would it be possible to achieve this with taints and tolerations? @hallowed-mouse-14616 do you remember if it’s possible to add tolerations by task type?
adding the concept of node pools explicitly is a bit tricky. flyte aims to be k8s native, but we still try to abstract the internals of k8s away from the user when possible and reasonable. also node pools aren’t a k8s construct right? they’re eks/gke constructs
you can apply labels/annotations to a given project or project-domain https://github.com/flyteorg/flyteidl/blob/master/protos/flyteidl/admin/matchable_resource.proto#L116
but i’m not sure how those tie into node pools
a
you could use taints and tolerations maybe too
s
Okay interesting ..
158 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by