Could it be that something has change within propeller when getting default platform resources? We’re trying to update from Flyte 1.3.0 to 1.6.0 at the moment and scheduler and worker pods of the

dask

plugin are unschedulable, as limits are not set properly. We’re thinking

executionMetadata.GetPlatformResources()

returns

nil

in the dask plugin, leading to limits not being set which is invalid with `ResourceQuota`’s in place. We do have a workaround of explicitly setting those limits for now but wanted to flag this.

This is possible, I know there's been a few updates. Very sorry about this, I will look into it.

Uhho

Great! I’ll also try to replicate, but will need to check whether my

dask

development setup still works

So this PR is the culprit. Basically, we were setting the resource overrides on the container to the containers resources, which is what made this approach work. However, now the resource overrides are completely empty (unless called from a workflow with

.with_overrides(...)

of course, so this is overriding the platform resources with an empty set. I think we can fix this by updating the

defaultResources

here to be

defaultContainerSpec.Resources

like is now done in the container helper code. Then we update this line to use

ResourceCustomizationModeMergeExistingResources

which will merge the container resources with overrides and then apply the defaults if none are set. I can take care of this, it was completely on us, but once I have a fix out would be great if you can help verify it's correct!

Great thank you! Do you mind if I fix the tests on your branch?

Thanks for checking, I was in meetings and planning on fixing these up. Ran into a little bit of a bug - will ping once ready!

Thank you! 🙇

Should be good, please take a look! I think there was a small misunderstanding about platformResources and how they work, basically the

limit

is a hard limit, which should cause the task to fail if it exceeds rather than using it as a limit.

Thank you! Reviewed ✅

Thanks so much Dan! Will there be a new flytepropeller release soon with the plugin dependency updated? Or perhaps there's a way to pin the plugin release version in the flytepropeller configs

I’ve opened a PR