Bernhard Stadlbauer
05/17/2023, 7:15 AMdask
plugin are unschedulable, as limits are not set properly.
We’re thinking executionMetadata.GetPlatformResources()
returns nil
in the dask plugin, leading to limits not being set which is invalid with `ResourceQuota`’s in place.
We do have a workaround of explicitly setting those limits for now but wanted to flag this.Dan Rammer (hamersaw)
05/17/2023, 1:28 PMKetan (kumare3)
Bernhard Stadlbauer
05/17/2023, 1:41 PMdask
development setup still worksDan Rammer (hamersaw)
05/17/2023, 2:51 PM.with_overrides(...)
of course, so this is overriding the platform resources with an empty set. I think we can fix this by updating the defaultResources
here to be defaultContainerSpec.Resources
like is now done in the container helper code. Then we update this line to use ResourceCustomizationModeMergeExistingResources
which will merge the container resources with overrides and then apply the defaults if none are set. I can take care of this, it was completely on us, but once I have a fix out would be great if you can help verify it's correct!Bernhard Stadlbauer
05/17/2023, 5:11 PMDan Rammer (hamersaw)
05/17/2023, 5:44 PMBernhard Stadlbauer
05/17/2023, 5:49 PMDan Rammer (hamersaw)
05/17/2023, 7:56 PMlimit
is a hard limit, which should cause the task to fail if it exceeds rather than using it as a limit.Bernhard Stadlbauer
05/22/2023, 12:48 PMDerek Yu
05/24/2023, 3:53 AMBernhard Stadlbauer
05/24/2023, 4:30 AM