🧵 I have read the documentation about Flyte agents and I feel like I understand how they simplify development of integrations with external services but I'm curious what the tradeoffs are with agents. At what point would I need to develop a plugin instead of going the agent route?

@clean-glass-36808 great question. the reason is performance

Thank you

@clean-glass-36808 we would love to learn how the experience has been with agents?

Its been okay, but I'll say that the general open source support has been good (especially compared to my experience working with the Armada folks). We have an Armada agent running that we're still evaluating and trying to understand what the limitations are (ie. dynamic tasks, map_task, etc). I think there are still some subtle gotchas in the docs depending on what the agent does. For example.. 1. If your agent is scheduling compute on some other system, you'll need to create a custom image (with pyflyte as a base) that includes your custom task packages. Otherwise the python code run by pyflyte-execute will fail. Default images will never work for these use cases. 2. On top of that, the custom image must not have the your custom agent packages. Otherwise when pyflyte-execute runs your task I believe it will try to run the agent logic locally on that binary as well. This was pretty confusing. Would also be good to understand best practices around logging on the agent and maybe error handling so things bubble up into the flyte UI nicely.

Ohh the 2 seems odd - and a bug. I think it should never load agents - cc @glamorous-carpet-83516 Runtime should bypass agents and go to the executor

I think the tl;dr is that we don't want to rely on k8s to queue workloads into a pending state think (think tens of thousands of tasks/jobs kicked off at once). We can't autoscale the cluster since we're not on public cloud. Additionally, we have some use cases for gang scheduling that I don't think the k8s scheduler is suitable for. Happy to invite my colleagues to share some lower level details

No makes sense - but I think if you use quota Flyte will back off till pods are available- makes sense

re: 1, It makes sense to me now but following along the flyte agent docs it wasn't obvious to me why I would need to make a custom image to run vanilla python logic in a task (ie. print hello world). It felt like the task decorator was more of a control plane level detail but yeah it makes sense now.

@clean-glass-36808 can you please help with doc update?

Yeah. I will validate #2 as well

Cc @glamorous-carpet-83516 / @damp-lion-88352 example skeleton to start different types of agents from would be awesome