https://flyte.org logo
#ask-the-community
Title
# ask-the-community
b

Bernhard Stadlbauer

05/17/2023, 7:15 AM
Could it be that something has change within propeller when getting default platform resources? We’re trying to update from Flyte 1.3.0 to 1.6.0 at the moment and scheduler and worker pods of the
dask
plugin are unschedulable, as limits are not set properly. We’re thinking
executionMetadata.GetPlatformResources()
returns
nil
in the dask plugin, leading to limits not being set which is invalid with `ResourceQuota`’s in place. We do have a workaround of explicitly setting those limits for now but wanted to flag this.
d

Dan Rammer (hamersaw)

05/17/2023, 1:28 PM
This is possible, I know there's been a few updates. Very sorry about this, I will look into it.
k

Ketan (kumare3)

05/17/2023, 1:37 PM
Uhho
b

Bernhard Stadlbauer

05/17/2023, 1:41 PM
Great! I’ll also try to replicate, but will need to check whether my
dask
development setup still works
d

Dan Rammer (hamersaw)

05/17/2023, 2:51 PM
So this PR is the culprit. Basically, we were setting the resource overrides on the container to the containers resources, which is what made this approach work. However, now the resource overrides are completely empty (unless called from a workflow with
.with_overrides(...)
of course, so this is overriding the platform resources with an empty set. I think we can fix this by updating the
defaultResources
here to be
defaultContainerSpec.Resources
like is now done in the container helper code. Then we update this line to use
ResourceCustomizationModeMergeExistingResources
which will merge the container resources with overrides and then apply the defaults if none are set. I can take care of this, it was completely on us, but once I have a fix out would be great if you can help verify it's correct!
OK, here is a PR for the associated issue.
b

Bernhard Stadlbauer

05/17/2023, 5:11 PM
Great thank you! Do you mind if I fix the tests on your branch?
d

Dan Rammer (hamersaw)

05/17/2023, 5:44 PM
Thanks for checking, I was in meetings and planning on fixing these up. Ran into a little bit of a bug - will ping once ready!
b

Bernhard Stadlbauer

05/17/2023, 5:49 PM
Thank you! 🙇
d

Dan Rammer (hamersaw)

05/17/2023, 7:56 PM
Should be good, please take a look! I think there was a small misunderstanding about platformResources and how they work, basically the
limit
is a hard limit, which should cause the task to fail if it exceeds rather than using it as a limit.
b

Bernhard Stadlbauer

05/22/2023, 12:48 PM
Thank you! Reviewed
d

Derek Yu

05/24/2023, 3:53 AM
Thanks so much Dan! Will there be a new flytepropeller release soon with the plugin dependency updated? Or perhaps there's a way to pin the plugin release version in the flytepropeller configs
b

Bernhard Stadlbauer

05/24/2023, 4:30 AM
I’ve opened a PR
65 Views