https://flyte.org
Join Slack
Is there a way I can modify default task tolerations in the helm charts values file? <https://github...
# flytekit
l
Is there a way I can modify default task tolerations in the helm charts values file? https://github.com/flyteorg/flyte/blob/master/charts/flyte/values.yaml
t
what’s the use-case? gpus?
more specifically, are you looking to add tolerations to everything? or only to certain task types?
f
Also for every container you can add default tolerations
We are working on adding a podTemplate (eventually per namespace), that will be used as the default template for all container tasks
l
My use case: we share a cluster with some other workloads (an mlflow tracking server instance, a webapp that uses the deployed models, …) and I don’t want these pods to use the expensive machines when no training is occurring, thus the taint. It is ok that all flyte pods by default have the toleration. With the provided links I was able to configure a default-toleation. Thanks for the guidance and really nice that this can be customized in such a fine-grained way :)
Is there a way to specify tolerations also for Spark tasks?
t
@gentle-kite-33075 do you know this by chance? tolerations for spark tasks?
f
but the spark plugin does not support that today. I think the spark plugin needs an upgrade. IMO it should be something like this
Copy code
@task(
config=Spark(), pod_template=Pod(),....)
def foo():
  ...
l
Actually setting this up in the spark helm chart itself is absolutely valid I think. Deploying the spark helm chart with a tolerations value fails atm but I will debug why.
Thanks for the hints!
@worried-farmer-76794
w
Thanks for the update. I think the 
tolerations
value of spark-operator helm chart doesn't work out of box. I will try to configure that one more time tomorrow.
l
@acceptable-policeman-57188
(Thread I mentioned in my dm.)
👀 1
178 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by