hello, (not sure if this is the right forum). I was wondering if flyte can handle realtime (while user is in session) data munging use cases. I see this was brought up in late 2021 in this thread https://github.com/flyteorg/flyte/discussions/1481

Realtime isn't a possibility today, Chetan. https://github.com/flyteorg/flyte/issues/3247

NP. FWIW - our use case is - We build data platform software focused on lifesciences / biotech vertical. We provide e2e software infra to run flyte orchestrated bioinformatics workflows. But quite often the data scientists want to slice and dice the results of these workflow result data frames in python/ r (for their secondary or tertiary scientific analysis/visualization. We were hoping to re-use and train them on a single flyte based programming model (async + sync). But since flyte doenst support it, this just means we have to train them on 2 separate programming models (for large async data crunching use flyte, for realtime in session adhoc type of crunching use a jupyter notebook or something similar perhaps) . If you ever need more use cases happy to provide.

Thanks for telling us the use-case, Chetan.

realtime in session adhoc type of crunching

Could you shed more light on this use-case of yours? I want to know how you expect Flyte to behave in such a scenario.

hi - here's a common scenario 1. End user visits one of our dashboard / visualization UIs and selects a chart in our webapp/ programmatic api. 2. Chart might take in some optional pre-defined inputs - (e.g. filter by date range, color/ group by field X, apply some cutoff / threshold on the data) 3. We pass these inputs and a handle to the raw underlying data to a user written flyte workflow. 4. The flyte workflow would execute user written python code that would apply any realtime / runtime filters, groupbys, etc on the data based on the user inputs and generate an output dataframe 5. Our runtime would use this dataframe to render the data in the chart 6. all of the above would happen while end user is waiting in the browser for their results

Gotcha. This isn't an ideal use-case of Flyte since it's more suitable for long-running workloads. I recommend you to keep tabs on the issue I shared to not miss any updates concerning the real-time scenarios.