has there been any thought given to realtime flyte applications? whether it be streaming or embedding flyte functions in a service? we have an use case where the overhead of spinning up flyte components provides some lag in our user experience, but we can't duplicate business logic into a web backend

cc: @Yee

what did you have in mind dylan could you elaborate please? what do you mean by streaming? been meaning to think more about streaming generally but haven’t really decided where to start yet. tagging @Haytham Abuelfutuh and @Dan Rammer (hamersaw)

@Dylan Wilder I'm also very interested in hearing about your specific use-case here. I know there has been some community effort in things like reactive workflows - if A happens, execute workflow B. This might be adapted / extended for things like batch processing streams, etc. Would be happy to discuss opportunities.

Does it take long for a task to finish @Dylan Wilder?

we have user interactive loops for which we currently wait for long processing calculations ("what if" scenario modeling). some of these models take a while to run and a long feedback loop is expected. others are quick (think mathematical formulas) and expectation is realtime feedback on parameter changes. so not the typical streaming pipeline application but we could definitely leverage it. the major bottleneck in some tasks is spool times, and in others we have use cases where we could run caculations for smaller sets of data for faster output but spool time would quickly become the bottleneck here as well

[flyte-core]

⭐ Create a new Flyte Core Feature issue: https://github.com/flyteorg/flyte/issues/new?assignees=&labels=enhancement%2Cuntriaged&template=feature_request.yaml&title=%5BCore+feature%5D+

@Dylan Wilder Unless this is done, I've created #3247 to track discussions and work about this feature request. Feel free to comment/add anything you may consider