Hi everyone, I've been trying to set pod tolerati...
# ask-the-community
Hi everyone, I've been trying to set pod tolerations for my spark tasks all day, and I'm not sure what exactly I'm doing wrong here. The tasks appear to be registering correctly, but when I execute them, I just don't see my tolerations pop up. I'm specifying the Spark task config, the container name and the pod template fields in the task.
Copy code
            tolerations=[V1Toleration(key="app", operator="Equal", value="flyte", effect="NoSchedule")],
Dunno whether this will display correctly as I had to write this on my phone. I've tried pretty much everything I can think of, from setting the spark configs, to creating the pod template on k8s and referencing it with the pod_template_name kwarg.
I am familiar with Databricks and Spark Config, but not familiar with spark in remote environment. Would you like to share more information? Such as the docs you have read, and any logs?
@Kevin Su Can help you
Sure thing. I referenced this PR (https://github.com/flyteorg/flyte/pull/4183) to ascertain whether this feature is implemented, based on what I'm seeing this ought to be the case. Log-wise it appears that the task is executing normally, however none of the tolerations are set upon closer inspection of the k8s yamls. I have set the node selector via the
configurations, but I have had no luck with doing the same with the
spark k8s configs. Either way this route shouldn't be necessary, considering the completed PR I included above.
@Kevin Su, I just checked the flyte console and it appears that tolerations are registered alongside the task, however, when task is executed they are not in the pod manifest.
@Kevin Su here are some screenshots
You can see the toleration when you describe the pod, but UI doesn’t show it, right? Feel like we forgot to update the entity in the database, so flyteconsole pulls wrong manifest
would you mind creating an issue for it? [flyte-bug]
No, I can't see the toleration in the pod description. I can see it in the UI, though. I'll create the issue yeah.
Thanks! you are using flyte v1.10, right
Yes I am. https://github.com/flyteorg/flyte/issues/4378 Here's the bug report.
which version of flytekit you are using?
I'm using 1.10. I can see that Andrew confirms that things are working on his end.