New RunLLM response generated: >>>*# Question* Hey we're looking for a workflow orchestrator to build some of our data pipelines. We were originally on Prefect, but encountered several bugs/limitations, so we're looking to switch. Currently, we're currently debating between dagster, restate, and flyte, but there aren't many resources to help us differentiate between the three. Some more context on our needs: * Backfill operation on several sources of data (not very demanding, just ~100gb) + some simple data transformations afterwards * Workflow oriented jobs to process our data further (example: file download/OCR/screenshots). This is currently very demanding. Each job varies in length (around 5-10 minutes), and we need to run 100s of them at the same time. These jobs need to run both as a backfill (totaling a few hundred thousand jobs over a few days), and afterwards continuously every day (~100-200) or so # Answer Flyte is well-suited for your needs, especially given its strengths in scalability, reproducibility, and handling large-scale workflows. Flyte can efficiently manage backfill operations and workflow-oriented jobs, leveraging Kubernetes for distributed processing and resource utilization. It supports backfill jobs through launch plans and can handle high degrees of parallelism, making it ideal for running hundreds of jobs simultaneously. For backfill operations, Flyte allows you to trigger launch plans across a range of dates, ensuring that each job runs on its own node and is evaluated at compile-time (source). Flyte's distributed scheduler can handle large-scale workflows, with capabilities to run millions of pipelines monthly (source). Additionally, Flyte's caching and versioning features can significantly speed up workflow execution and reduce costs (source). Would you like more detailed comparisons or specific examples of how Flyte can be implemented for your use case? Sources: - Flyte Backfill and Look-back - Flyte vs. Other Workflow Managers - Flyte Introduction Slack Link https://.slack.com/archives/CP2HDHKE1/p1728588135.417089 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.