Is there a way I can modify default task toleratio...
# flytekitf
Fabio Grätz
02/07/2022, 7:05 PMIs there a way I can modify default task tolerations in the helm charts values file?
https://github.com/flyteorg/flyte/blob/master/charts/flyte/values.yaml
y
Yee
02/07/2022, 7:44 PM
what’s the use-case? gpus?
Yee
02/07/2022, 7:52 PM
more specifically, are you looking to add tolerations to everything? or only to certain task types?
Yee
02/07/2022, 7:58 PM
Yee
02/07/2022, 7:58 PM
Yee
02/07/2022, 7:58 PM
k
Ketan (kumare3)
02/07/2022, 10:17 PM
Also for every container you can add default tolerations
Ketan (kumare3)
02/07/2022, 10:17 PM
We are working on adding a podTemplate (eventually per namespace), that will be used as the default template for all container tasks
f
Fabio Grätz
02/08/2022, 10:37 AMMy use case: we share a cluster with some other workloads (an mlflow tracking server instance, a webapp that uses the deployed models, …) and I don’t want these pods to use the expensive machines when no training is occurring, thus the taint. It is ok that all flyte pods by default have the toleration.
With the provided links I was able to configure a default-toleation. Thanks for the guidance and really nice that this can be customized in such a fine-grained way :)
Fabio Grätz
02/08/2022, 3:38 PMIs there a way to specify tolerations also for Spark tasks?
y
Yee
02/08/2022, 10:34 PM
@Anmol do you know this by chance? tolerations for spark tasks?
k
Ketan (kumare3)
02/08/2022, 11:17 PM
@Fabio Grätz there is a way to add tolerations for spark - https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/pkg/apis/sparkoperator.k8s.io/v1beta2/types.go#L440
Ketan (kumare3)
02/08/2022, 11:18 PM
but the spark plugin does not support that today. I think the spark plugin needs an upgrade.
IMO it should be something like this
Copy code
@task(
config=Spark(), pod_template=Pod(),....)
def foo():
...f
Fabio Grätz
02/10/2022, 11:29 AMActually setting this up in the spark helm chart itself is absolutely valid I think. Deploying the spark helm chart with a tolerations value fails atm but I will debug why.
Fabio Grätz
02/10/2022, 11:29 AMThanks for the hints!
Fabio Grätz
03/14/2022, 4:07 PM@Emirhan Karagül
e
Emirhan Karagül
03/14/2022, 4:18 PMThanks for the update. I think the
tolerationsf
Fabio Grätz
04/10/2022, 2:41 PM@katrina
Fabio Grätz
04/10/2022, 2:41 PM(Thread I mentioned in my dm.)
👀 1
168 Views