Hi Flyte team, Looking at the flytekit plugins for KF jobs: Pytorch, Tensorflow and MPI. None of them seems to provide resource configs for workers/launcher separately. Wondering why is this the case? Does Flyte has some internal logic to “smartly” assign resources across different type of pods?

you’re right. we don’t have the separate resource config for the worker, so now both master and workers will use same resource. we’d like to add new config for workers, and it will be included in the next release I think.

@Yubo Wang do you know if there is currently an issue for this? If not, would you be willing to file one? This seems like a very simple fix - maybe a community member could pick it up!

@Dan Rammer (hamersaw) let me do some search, if not I will file one

[flyte-core]

⭐ Create a new Flyte Core Feature issue: https://github.com/flyteorg/flyte/issues/new?assignees=&labels=enhancement%2Cuntriaged&template=feature_request.yaml&title=%5BCore+feature%5D+

^^ @Yubo Wang for your convenience 🙂

https://github.com/flyteorg/flyte/issues/3308 just got time to circle back on this. Please take a look. I am also willing to do the implementation if possible.

@Yubo Wang thanks for filing this! I left a brief comment on the issue outlining how Flyte currently handles this on the Dask plugin. It would be fantastic if this is something you're willing to contribute! The PRs for the recent add of the Dask plugin would be a great place to start. Let me know if we can help clarify anything.