Hi All, I dont know if this is the appropriate channel to ask this question but here we go... I am evaluating to implement Flyte for our company and i am almost technically convinced to do so, but there is one thing that push me back, which is streaming support. I have read that in the 1,5 version F*lyte* is supporting File Streaming, but when i mean streaming support i am referring to the capability of connecting some stream native source like Kafka, Nats or MQTT to continually run a workflow when a new event appears. Is there a plan in the roadmap in the near future to implement something like this or for some technical reason it is better to not use flyte for stream data pipelines? It seems to me that everything is batch based. Should i implement my self as a Task Plugin? Is there a deep dive documentation on how to do it properly? This last question is more related to the fact that stream processing requires some back pressure / throttling mechanism in order to not saturate the consumer. Thanks in advance and sorry if this is not the correct channel to ask

@Juan Pablo Royo Sales thank you for the detailed question. So you can trigger Flyte workflows on an event using the api. The new agent framework can support polling an event source too. But having a workflow like a dataflow stream is currently not in scope, though we do plan to support something called as streamliterals - these are literal values that are streamed and so that tasks can continue to be run and be warm, but so far we have not planned it for this year

Thanks for the quick answer

@Juan Pablo Royo Sales but, we would love to discuss more, as to what usecases you see and how would you want to interaction model to be