Hi Community! We use Flyte with the Databricks (GO) plugin, running Dynamic workflows. In some cases, failed dynamic nodes would get stuck in propeller. Even though the nodes are marked as

FAILED

/

ABORTED

in the Flyte UI, propeller continues to poll the Databricks job status for these "stale" nodes until propeller restart. Are you aware of this issue? cc @gentle-state-35322 @full-toddler-5766 @careful-holiday-56196

Cc @glamorous-carpet-83516 this looks like a bug right

which version of propeller you are using

propeller version is v1.10.6

@full-toddler-5766 / @aloof-painting-18735 it seems this was a bug that was fixed prior to 1.10.6 - https://github.com/flyteorg/flyte/commit/da77476ecbb8b183612405e9f3e3f7117d67d2a8 So i am confused.

and also was fixed in this pr. https://github.com/flyteorg/flytestdlib/pull/164

@full-toddler-5766 / @aloof-painting-18735 / @gentle-state-35322 1. the resource manager implementations are available here. 2. You can implement a Inmemory or Redis resource manager that uses secure connections - here 3. This is the logic of checking the current set quotas 4. This is where the quotas are configured 5. If you have redis configured this should work To create the configuration. 1. You will have to set the type of resource manager at propeller config level 2. Once you have that, for the databricks plugin - you will need to create the expected resource quotas. 3. ResourceQuotas is simply a map of

namespace: quota value (int)

4. ResourceNamespace is just a string 5. This explains how resource manager works 6. Default namespace is already configured 7. Currently it defaults to 100 project level and 50 per namespace I think you should be able to just simply set the defaults to a low value and it should work if you have redis configured. Let me figure out if this recommendation by @high-park-82026 is indeed accurate or should we set the

defaults

Not sure if the resourcequota is used. cc @tall-lock-23197 and @glamorous-carpet-83516 this might be useful to test with Agent as well and then document

@freezing-airport-6809 Thanks for the detailed instructions, let us review.