Hello Flyte team! We are using Flyte to schedule spark driver and executor pods. The problem is that the tolerations we use somehow don't get applied to these pods. They however work fine for all the other pods. Example tolerations on a spark pod (driver and executor the same):

Tolerations:                 <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                             <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s

Example tolerations on another pod (also flyte created).

Tolerations:                 <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                             <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
                             <http://nvidia.com/gpu=present:NoSchedule|nvidia.com/gpu=present:NoSchedule>
                             <http://nvidia.com/gpu:NoSchedule|nvidia.com/gpu:NoSchedule> op=Exists
                             purpose=compute:NoSchedule

We already tried to restart the node pool. Is this somehow an expected behavior? Flytekit version:

1.1.0

Hi @prehistoric-kangaroo-27449, are these tolerations specified in the k8s plugin configuration within flytepropeller?

@worried-farmer-76794 cc

Aah, @worried-farmer-76794 this was started by @late-pencil-3873 do you want to Help Push it over

I'd be down if @late-pencil-3873 has time. Do you guys already have a design proposal to make this work?

As @hallowed-mouse-14616 commented there is a PR

@worried-farmer-76794 so with the above PR we were attempting to upgrade the k8s-on-spark-operator as well (figured lets just update everything during configuration changes). However, the newer version imposed some dependencies, specifically k8s version, that because of the way this is compiled broke other plugins - this will be fixed soon. So to wrap this up we just need to scope down the PR to handle only the tolerations (and hopefully other existing fields) without attempting to upgrade the k8s-on-spark-operator.