https://flyte.org logo
f

Fabio Grätz

02/07/2022, 7:05 PM
Is there a way I can modify default task tolerations in the helm charts values file? https://github.com/flyteorg/flyte/blob/master/charts/flyte/values.yaml
y

Yee

02/07/2022, 7:44 PM
what’s the use-case? gpus?
more specifically, are you looking to add tolerations to everything? or only to certain task types?
k

Ketan (kumare3)

02/07/2022, 10:17 PM
Also for every container you can add default tolerations
We are working on adding a podTemplate (eventually per namespace), that will be used as the default template for all container tasks
f

Fabio Grätz

02/08/2022, 10:37 AM
My use case: we share a cluster with some other workloads (an mlflow tracking server instance, a webapp that uses the deployed models, …) and I don’t want these pods to use the expensive machines when no training is occurring, thus the taint. It is ok that all flyte pods by default have the toleration. With the provided links I was able to configure a default-toleation. Thanks for the guidance and really nice that this can be customized in such a fine-grained way :)
Is there a way to specify tolerations also for Spark tasks?
y

Yee

02/08/2022, 10:34 PM
@Anmol do you know this by chance? tolerations for spark tasks?
k

Ketan (kumare3)

02/08/2022, 11:17 PM
but the spark plugin does not support that today. I think the spark plugin needs an upgrade. IMO it should be something like this
Copy code
@task(
config=Spark(), pod_template=Pod(),....)
def foo():
  ...
f

Fabio Grätz

02/10/2022, 11:29 AM
Actually setting this up in the spark helm chart itself is absolutely valid I think. Deploying the spark helm chart with a tolerations value fails atm but I will debug why.
Thanks for the hints!
@Emirhan Karagül
e

Emirhan Karagül

03/14/2022, 4:18 PM
Thanks for the update. I think the
tolerations
value of spark-operator helm chart doesn't work out of box. I will try to configure that one more time tomorrow.
f

Fabio Grätz

04/10/2022, 2:41 PM
@katrina
(Thread I mentioned in my dm.)
👀 1
3 Views