https://flyte.org
Join Slack
Is there a way to automatically taint a few node pools to allow gpu access upon creation? Like <this...
# flyte-support
d
Is there a way to automatically taint a few node pools to allow gpu access upon creation? Like this but through the values.yaml when helm installing? Also what should the key and value be for the resource tolerations?
t
can you elaborate a bit?
what do you mean by automatically taint?
setting taints on node pools is not something that’s done by flyte. that will have to be done by whoever is typically in charge of setting up the aws/eks resources.
once the taints are set, the tolerations for the gpu resource type should just be whatever you set as the taint.
d
So I can set the tolerations on the node pools by doing this in values.yaml
Copy code
k8s:
    plugins:
      # -- Configuration section for all K8s specific plugins [Configuration structure](<https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/pluginmachinery/flytek8s/config>)
      k8s:
        default-env-vars: []
        #  DEFAULT_ENV_VAR: VALUE
        default-cpus: 100m
        default-memory: 100Mi
        resource-tolerations:
          - <http://nvidia.com/gpu|nvidia.com/gpu>:
            - key: "flyte/gpu"
              operator: "Equal"
              value: "dedicated"
              effect: "NoSchedule"
But in order for a node pool to have a taint that matches these tolerations, I would need to taint them after helm installation. Is there a way to avoid doing the manual step of looking up node pools and setting the taint? It would be ideal if the node pool creator would apply taints to a few node pools.
t
how are you creating the node pools?
and yes that’s correct - it’s a separate step
but note that this is just text (well a map of text)
basically it’s saying, for tasks that use that resource, apply these tolerations.
if you’re using terraform or something to create the node pools i’m sure it’s possible to add the taints at creation time.
flyte itself does not create node pools, it merely uses them
d
okay i understand now. thank you. I thought flyte created node pools since there are some default nodes under the development namespace.
175 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by