New RunLLM response generated: >>>*# Question* Hi everyone, we have a use-case where we want to create workflows on the fly, possibly driven by a UX builder. This can mean possible 1000's of custom workflows. We can handle the cruft of converting JSON -> the python sdk for flyte, and registering the new workflows. What we are unsure of is if Flyte can handle these many custom workflows. Is this a use-case thats been tried before? And possibly is there another way instead of JSON -> PythonSDK? Thanks a lot for your help. # Answer Flyte can handle a large number of custom workflows, as it is designed for scalability and can manage thousands of concurrent workflows. According to Flyte's documentation, Flyte can scale to millions of executions and containers, making it suitable for handling thousands of custom workflows. For creating workflows dynamically, Flyte supports dynamic workflows, which can be constructed at runtime based on inputs. This feature is particularly useful for scenarios where the workflow structure is not known ahead of time and needs to be generated dynamically. You can read more about dynamic workflows in Flyte here. Regarding the conversion from JSON to Python SDK, while Flyte primarily uses Python for defining workflows, you can automate the generation of Python code from JSON definitions. However, Flyte does not natively support direct JSON-to-workflow conversion without using the Python SDK. You would need to implement a custom solution to parse JSON and generate the corresponding Python code for Flyte workflows. Would you like more details on any specific aspect of this process? Sources: - Flyte Scalability - Dynamic Workflows in Flyte Slack Link https://flyte-org.slack.com/archives/CP2HDHKE1/p1722887153.204129 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.