When subworkflows/tasks are not run because the exceed the resource quota, shouldn't they automatically start running once enough resources are available again or the limits were increased?
Right now the tasks stay in running, but nothing happens...
01/05/2023, 6:03 PM
yes. they should. do you see that not happening?
that issue isn’t entirely related right?
that bug is an admin bug… when it creates the pod, admin isn’t respecting the project/domain level setting, but is creating the pod with the base Admin settings instead
but once the Pod is created, with whatever resources it’s created with, that behavior should take over
01/05/2023, 6:06 PM
I updated the cluster-resource via flytectl first, that didn't work as per the bug.
I then increased the defaults in the my helm chart values and updated that, which restarted flyteadmin, then they correctly were launched automatically
Didn't see that at first, because in console it all shows as running, and only in the details you see that the actual status is queued or so
01/05/2023, 6:08 PM
got it, yeah the statuses are a bit misleading
01/05/2023, 6:11 PM
Any plans to improve that as well? The correct information seems to be there....
01/05/2023, 6:14 PM
reporting and observability is definitely a topic high on the list in general. mind filing an issue for it though with some screenshots if you see it again?