I have the question about the Flyte Deepdive architecture, I have read about the Flyte that Flyte support natively scheduler which is used information about cronSchedule( user have to defind in each lauchplan). I'm wandering if multi user trigger the task by using Flytekit, What Flyte Schedule will do in this problem.
11/20/2023, 4:48 AM
@Ryuu, could you elaborate more? after you define a schedule and activate it, it's going to run according to the specified cadence. are you asking what happens if the launch plan is activated multiple times? if you've a schedule associated with your workflow, it can be activated and deactivated once. you can, however, have multiple schedules associated with a single workflow.
11/20/2023, 8:19 AM
@Samhita Alla I have read about the native scheduler and i'm understanding that the flyte propeller with scheduler optimize all launchplan natively (which contain sth like sched = CronSchedule("* * ? * * *") ).
But if i trigger the task directly by trigger on UI, or many user trigger 1 reference task through Flyte kit. Then all this execution is not Schedule before, and it was suddenly requested to exec. Then FlytePropeller will deal with this situation for optimizing, or just simplely queue.
I'm thinking about the module that multi users will push the own data to Cloud. This Execution is not have CronSchedule, How The Flyte system deal with multi this task execution at one specific time.
Have you had any idea for me to implement the module like this. Because the data stream maybe very very large.
11/20/2023, 10:56 AM
not all launch plans have a schedule, but only the ones you define a schedule for! so when you trigger a workflow from the UI, it should just work.
11/20/2023, 12:53 PM
@Samhita Alla If so, if multiple users trigger multiple tasks at a time, those tasks will be scheduled and executed by k8s, right? In essence, they still need to be scheduled because resources are limited.
11/20/2023, 1:18 PM
yes, depending on the resources available, they'll be queued or executed. but it's gonna work!