Hi everyone, I've been trying to set pod tolerations for my spark tasks all day, and I'm not sure what exactly I'm doing wrong here. The tasks appear to be registering correctly, but when I execute them, I just don't see my tolerations pop up. I'm specifying the Spark task config, the container name and the pod template fields in the task.

@task(
    task_config=Spark(...),
    container_image=...,
    pod_template=PodTemplate(
        ...,
        pod_spec=V1PodSpec(
            ...
            tolerations=[V1Toleration(key="app", operator="Equal", value="flyte", effect="NoSchedule")],
         ),
    ),
)

Dunno whether this will display correctly as I had to write this on my phone. I've tried pretty much everything I can think of, from setting the spark configs, to creating the pod template on k8s and referencing it with the pod_template_name kwarg.

I am familiar with Databricks and Spark Config, but not familiar with spark in remote environment. Would you like to share more information? Such as the docs you have read, and any logs?

Sure thing. I referenced this PR (https://github.com/flyteorg/flyte/pull/4183) to ascertain whether this feature is implemented, based on what I'm seeing this ought to be the case. Log-wise it appears that the task is executing normally, however none of the tolerations are set upon closer inspection of the k8s yamls. I have set the node selector via the

spark.kubernetes

configurations, but I have had no luck with doing the same with the

podTemplateFile

spark k8s configs. Either way this route shouldn't be necessary, considering the completed PR I included above.

You can see the toleration when you describe the pod, but UI doesn’t show it, right? Feel like we forgot to update the entity in the database, so flyteconsole pulls wrong manifest

🐞 Create a new Flyte Bug issue: https://github.com/flyteorg/flyte/issues/new?assignees=&labels=bug%2Cuntriaged&template=bug_report.yaml&title=%5BBUG%5D+

No, I can't see the toleration in the pod description. I can see it in the UI, though. I'll create the issue yeah.

Thanks! you are using flyte v1.10, right

Yes I am. https://github.com/flyteorg/flyte/issues/4378 Here's the bug report.

which version of flytekit you are using?

I'm using 1.10. I can see that Andrew confirms that things are working on his end.