Is there a way I can modify default task toleratio...
# flytekit
Is there a way I can modify default task tolerations in the helm charts values file?
what’s the use-case? gpus?
more specifically, are you looking to add tolerations to everything? or only to certain task types?
Also for every container you can add default tolerations
We are working on adding a podTemplate (eventually per namespace), that will be used as the default template for all container tasks
My use case: we share a cluster with some other workloads (an mlflow tracking server instance, a webapp that uses the deployed models, …) and I don’t want these pods to use the expensive machines when no training is occurring, thus the taint. It is ok that all flyte pods by default have the toleration. With the provided links I was able to configure a default-toleation. Thanks for the guidance and really nice that this can be customized in such a fine-grained way :)
Is there a way to specify tolerations also for Spark tasks?
@Anmol do you know this by chance? tolerations for spark tasks?
but the spark plugin does not support that today. I think the spark plugin needs an upgrade. IMO it should be something like this
Copy code
config=Spark(), pod_template=Pod(),....)
def foo():
Actually setting this up in the spark helm chart itself is absolutely valid I think. Deploying the spark helm chart with a tolerations value fails atm but I will debug why.
Thanks for the hints!
@Emirhan Karagül
Thanks for the update. I think the
value of spark-operator helm chart doesn't work out of box. I will try to configure that one more time tomorrow.
(Thread I mentioned in my dm.)
👀 1