elegant-petabyte-32634
06/01/2022, 3:57 PM# -- Kubernetes specific Flyte configuration
k8s:
plugins:
# -- Configuration section for all K8s specific plugins [Configuration structure](<https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/pluginmachinery/flytek8s/config>)
k8s:
default-env-vars: []
# DEFAULT_ENV_VAR: VALUE
default-cpus: 100m
default-memory: 100Mi
resource-tolerations:
- <http://nvidia.com/gpu|nvidia.com/gpu>:
- key: "key1"
operator: "Equal"
value: "value1"
effect: "NoSchedule"
and I applied that with helm and also tried restarting the Flyte pods (kubectl rollout restart deploy), but the pods that get started by Flyte workflows don't get these tolerations (although they do get a default nvidia.com/gpu "exists" toleration regardless of my addition above). Anything I'm doing wrong?elegant-petabyte-32634
06/01/2022, 4:01 PMacceptable-policeman-57188
elegant-petabyte-32634
06/01/2022, 5:38 PMelegant-petabyte-32634
06/01/2022, 5:40 PMfrom flytekit import task, workflow, Resources
@task(
requests=Resources(gpu="1", cpu="2"),
limits=Resources(mem="8Gi"),
)
def test_gpu():
...
@workflow
def wf():
test_gpu()
# pyflyte run --remote -p flytesnacks -d development testgpu.py wf
acceptable-policeman-57188
elegant-petabyte-32634
06/01/2022, 5:47 PMacceptable-policeman-57188
do get a default nvidia.com/gpu "exists" toleration regardless of my addition above).is this only for the test_gpu task pod or all pods?
elegant-petabyte-32634
06/01/2022, 5:53 PMelegant-petabyte-32634
06/01/2022, 5:54 PMacceptable-policeman-57188
kubectl -n flyte port-forward deploy/flytepropeller 10254
and going to http://localhost:10254/configelegant-petabyte-32634
06/01/2022, 6:00 PMelegant-petabyte-32634
06/01/2022, 6:03 PMacceptable-policeman-57188
acceptable-policeman-57188
elegant-petabyte-32634
06/01/2022, 6:06 PMacceptable-policeman-57188
acceptable-policeman-57188
elegant-petabyte-32634
06/01/2022, 6:07 PMelegant-petabyte-32634
06/01/2022, 6:17 PMacceptable-policeman-57188