@ancient-wolf-19325 is there a practical limit on how much data a task/workflow can process?

Yes, there are practical limits on the amount of data a task or workflow can process in Flyte. These limits are primarily due to constraints in the underlying Kubernetes infrastructure and Flyte's use of etcd for state storage. 1. etcd Storage Limit: Flyte uses etcd to store workflow execution state, which has a limit of 1.5MB for object sizes. This can restrict the size of workflows and the amount of metadata that can be stored (source). 2. FlyteCopilot Download Limit: There is a limit on the size of data that FlyteCopilot can download, which was previously set to 2MB but can be configured to be unlimited (source). Would you like more detail on any specific aspect? Sources: - https://discuss.flyte.org/t/16094488/lets-say-you-have-a-task-that-you-need-to-run-100-000-times-/ - https://github.com/flyteorg/flyte/issues/1251 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 what is the best practice for processing large streams of data in a task?

For processing large streams of data in a Flyte task, the best practices include: 1. Streaming Data: Use streaming to handle large data efficiently without loading it all into memory. Flyte is working on enhancing its data subsystem to support streaming (source). 2. Shared Volumes: Utilize shared volumes like EFS, Lustre, or FSx to avoid repeatedly uploading and downloading large datasets between tasks (source). 3. Map Tasks: Use map tasks for parallel processing of data chunks, which can improve performance by distributing the workload across multiple nodes (source). Would you like more detail on any specific aspect? Sources: - https://discuss.flyte.org/t/9660957/hello-i-m-at-a-point-with-flyte-now-where-i-m-asking-myself-/ - https://github.com/flyteorg/flytesnacks/blob/master/examples/advanced_composition/advanced_composition/map_task.py 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.