https://flyte.org logo
#ask-the-community
Title
# ask-the-community
j

Jan Costandius

11/07/2023, 2:14 PM
Hi everyone, I've been trying to set pod tolerations for my spark tasks all day, and I'm not sure what exactly I'm doing wrong here. The tasks appear to be registering correctly, but when I execute them, I just don't see my tolerations pop up. I'm specifying the Spark task config, the container name and the pod template fields in the task.
Copy code
@task(
    task_config=Spark(...),
    container_image=...,
    pod_template=PodTemplate(
        ...,
        pod_spec=V1PodSpec(
            ...
            tolerations=[V1Toleration(key="app", operator="Equal", value="flyte", effect="NoSchedule")],
         ),
    ),
)
Dunno whether this will display correctly as I had to write this on my phone. I've tried pretty much everything I can think of, from setting the spark configs, to creating the pod template on k8s and referencing it with the pod_template_name kwarg.
l

L godlike

11/07/2023, 2:56 PM
I am familiar with Databricks and Spark Config, but not familiar with spark in remote environment. Would you like to share more information? Such as the docs you have read, and any logs?
@Kevin Su Can help you
j

Jan Costandius

11/07/2023, 3:09 PM
Sure thing. I referenced this PR (https://github.com/flyteorg/flyte/pull/4183) to ascertain whether this feature is implemented, based on what I'm seeing this ought to be the case. Log-wise it appears that the task is executing normally, however none of the tolerations are set upon closer inspection of the k8s yamls. I have set the node selector via the
spark.kubernetes
configurations, but I have had no luck with doing the same with the
podTemplateFile
spark k8s configs. Either way this route shouldn't be necessary, considering the completed PR I included above.
@Kevin Su, I just checked the flyte console and it appears that tolerations are registered alongside the task, however, when task is executed they are not in the pod manifest.
@Kevin Su here are some screenshots
k

Kevin Su

11/07/2023, 5:59 PM
You can see the toleration when you describe the pod, but UI doesn’t show it, right? Feel like we forgot to update the entity in the database, so flyteconsole pulls wrong manifest
would you mind creating an issue for it? [flyte-bug]
j

Jan Costandius

11/07/2023, 7:04 PM
No, I can't see the toleration in the pod description. I can see it in the UI, though. I'll create the issue yeah.
k

Kevin Su

11/07/2023, 7:29 PM
Thanks! you are using flyte v1.10, right
j

Jan Costandius

11/07/2023, 7:46 PM
Yes I am. https://github.com/flyteorg/flyte/issues/4378 Here's the bug report.
k

Kevin Su

11/07/2023, 11:14 PM
which version of flytekit you are using?
j

Jan Costandius

11/08/2023, 5:29 AM
I'm using 1.10. I can see that Andrew confirms that things are working on his end.
7 Views