Robin Kahlow
06/01/2022, 3:57 PM# -- Kubernetes specific Flyte configuration
k8s:
plugins:
# -- Configuration section for all K8s specific plugins [Configuration structure](<https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/pluginmachinery/flytek8s/config>)
k8s:
default-env-vars: []
# DEFAULT_ENV_VAR: VALUE
default-cpus: 100m
default-memory: 100Mi
resource-tolerations:
- <http://nvidia.com/gpu|nvidia.com/gpu>:
- key: "key1"
operator: "Equal"
value: "value1"
effect: "NoSchedule"
and I applied that with helm and also tried restarting the Flyte pods (kubectl rollout restart deploy), but the pods that get started by Flyte workflows don't get these tolerations (although they do get a default nvidia.com/gpu "exists" toleration regardless of my addition above). Anything I'm doing wrong?katrina
Robin Kahlow
06/01/2022, 5:38 PMfrom flytekit import task, workflow, Resources
@task(
requests=Resources(gpu="1", cpu="2"),
limits=Resources(mem="8Gi"),
)
def test_gpu():
...
@workflow
def wf():
test_gpu()
# pyflyte run --remote -p flytesnacks -d development testgpu.py wf
katrina
Robin Kahlow
06/01/2022, 5:47 PMkatrina
do get a default nvidia.com/gpu "exists" toleration regardless of my addition above).is this only for the test_gpu task pod or all pods?
Robin Kahlow
06/01/2022, 5:53 PMkatrina
kubectl -n flyte port-forward deploy/flytepropeller 10254
and going to http://localhost:10254/configRobin Kahlow
06/01/2022, 6:00 PMkatrina
Robin Kahlow
06/01/2022, 6:06 PMkatrina
Robin Kahlow
06/01/2022, 6:07 PMkatrina