@Byron Hsu I saw your contributions to langchain. Would love to integrate more of langchain and Flyte. Do you have ideas on it

@Ketan (kumare3) I think langchain is just a high-level python library. It can just work with flyte out-of-the-box. Adding a one more layer (like langchain plugin) might get things complicated IMO. Just creating some examples to demonstrate how flyte work with langchain is sufficient

i know, but i think would it help with having orchestration around some stuff - i guess but you are right one could simply write a workflow where the

loader

chunking

etc in langchain is done as a task and the actuall call to llms is done in a separate task

yeah that’s what i did in flyteGPT. LangChain is mostly for inference. I think users can do “bulk inference” with LangChain on Flyte, and move on to serving if the result is good.

whats an airbyte langchain plugin - huh?

im thinking if we can build a custom chain consisting of flyte tasks, which is basically a flyteworkflow. https://python.langchain.com/en/latest/modules/chains/generic/custom_chain.html users can inherits or use this chain if they want to run the chain as flyte workflow on flyte

I think a Flyte example (literally Byron’s FlyteGPT) is the path of least resistance, if the goal is to get more exposure for Flyte. In terms of a custom chain/deeper integration, I think the promising directions that will provide value to users are: • the data preprocessing steps (document loading, chunking, indexing) taking advantage of Flyte’s caching • batch inference for use case that don’t have low latency requirement. Basically if we can get LangChain abstractions to run on Flyte natively that would be very interesting. Haven’t thought this through, but something like:

chain = FlyteChain(some_langchain_chain, remote=FlyteRemote(...))
chain.run(...)  # the steps in this some_langchain_chain will just run on a Flyte cluster

Definitely think

FlyteChain would be very cool and on the LLM hype, but not very useful in reality 😂😂

Why not useful in reality?

what is agent framework?

i mean the latency to launch the tasks might be longer than running in a single program

Not every task needs to be launched, better synchronous plug-ins that can be run directly as a

is it like tasks act as a server so it can be reused?

Yup

But the tasks have to share the same deps if they want to run on the same pod?

Ya like an agent

That could speed up many workflows a lot

Yup

What’s that