@ancient-wolf-19325 why is workflow/task latency so high?

High workflow/task latency in Flyte can be attributed to several factors, including Kubernetes queuing latency, transition latency between node executions, and Flyte's orchestration overhead. According to the Flyte documentation, queuing latency is the time taken by Kubernetes to start the pod, which can be influenced by available resources and quotas. Transition latency involves the time between successive node executions, including gathering outputs and inputs, and sending events to FlyteAdmin. Additionally, performance benchmarking indicates that Flyte's orchestration operations, such as creating/scheduling Pods and managing container runtimes, contribute to overhead. This overhead varies by node type and can be significant, especially in complex workflows with dynamic tasks or high parallelism. Would you like more detail on specific latency types or potential optimizations? Sources: - Flyte Execution Timeline - Flyte Performance Benchmarking 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.