https://flyte.org logo
#ask-the-community
Title
# ask-the-community
c

Christoph Neumaier

03/27/2024, 1:13 PM
Hi everybody, I am trying to understand the different integration options available in flyte. What exactly is the difference between a "flyte agent" and an "External Service Backend Plugin"? In the docs there is an agent documentation like the "Memory Machine Cloud" that executes flyte tasks externally. That could have been implemented as external plugin as well. Can somebody explain to me what the optimal use case for each option is and what the limitaions are? Thank you a lot for taking the time 🙌
d

David Espejo (he/him)

03/27/2024, 1:31 PM
Hi @Christoph Neumaier and welcome to the Flyte community. Well, I've had a similar question, so let me try to answer from what I know (others can jump and correct/expand). Every interaction that Flyte performs with any outside system (by outside I mean even Kubernetes) uses a plugin. Let's say you have a workflow and one of the tasks uses MMCloud, for example. With a regular plugin, a K8s Pod would be created for each task, so you'd have a Pod whose sole purpose is to execute the request to the MMCloud. Doesn't sound much efficient in terms of resource consumption and overall latency right? Also, the person/team who wrote that plugin has to be not only proficient in Go (not typical in MLOps/Data teams) but to know intimately how to integrate that plugin into Propeller, Flyte's execution engine. In the other hand, if you use, say, the MMCloud Agent, your Task will contact a "Connection pool" on a long-running service (one that could even run multiple different Agents concurrently) so your resource consumption footprint is significantly reduced and the overall performance could be better; so much so you can even have Synchronous Agents. And, finally, Agents are written in Python (potentially it could be other languages too as it uses protobuf for serialization). Does that help?
c

Christoph Neumaier

03/27/2024, 1:34 PM
Hi @David Espejo (he/him) thank you for your response. Your explanation make sense in every way. Sounds like the agent approach is the more efficient one. Thank you for clarification.
Maybe you can even answer another question i have: Are there capabilities in flyte and the agent integrations for bidirectional communication before a task is being executed? Lets say you want to execute a task externally, like with MMCloud, and you need to confirm some seetings or anything that the agent came up with and the user needs to confirm that before the task gets finally executed
d

David Espejo (he/him)

03/27/2024, 1:54 PM
something like Human in the Loop? For an async agent you can implement a
get
method but gating the execution on user input would require an additional feature https://docs.flyte.org/en/latest/user_guide/advanced_composition/waiting_for_external_inputs.html is this what you're looking for?
k

Ketan (kumare3)

03/27/2024, 2:15 PM
@Christoph Neumaier I want to add couple points quick external service backend plugins and agents are the same thing. We are deprecating external service backend plugins in favor of agents. Check out this blog https://www.union.ai/blog-post/flyte-agents-framework Now to answer second you can wait using wait for input tasks, but you will have to have a three step workflow. Step one check something with the external service, then run a task to wait for approval and then finally use the agent to do something on the external service. It is possible to offer the composition as a simple method for your users. And example is sagemaker deployment https://github.com/flyteorg/flytekit/blob/fa2aa0b215cd2fca3d90d8de92e41c6fadb88ed6/plugins/flytekit-aws-sagemaker/flytekitplugins/awssagemaker_inference/workflow.py#L35 What this does is creates a workflow that can be used in another workflow
Cc @David Espejo (he/him) ^
c

Christoph Neumaier

03/27/2024, 2:47 PM
@David Espejo (he/him) and @Ketan (kumare3) thank you both for the provided input. I will take a more in depth look at the supplied solutions from both of you and report back after i understood them completely.
2 Views