https://flyte.org logo
Join Slack
Powered by
New RunLLM response generated: >>>*# Question* We...
# runllm-flyte-trial
a
New RunLLM response generated: >>>*# Question* We are seeing many messages such as this one right after deployment:
Copy code
failed to create execution create request %+v due to %vproject:"redacted" domain:"development" name:"ffc074c35db9fa02c000" spec:<launch_plan:<resource_type:LAUNCH_PLAN project:"redacted" domain:"development" name:"redacted_launchplan" version:"d0f89907dc495858c49c882cb05b5d564ba859ee" > metadata:<mode:SCHEDULED scheduled_at:<seconds:1718884800 > > > inputs:<>  rpc error: code = Internal desc = failed to create workflow in propeller the server could not find the requested resource (post <http://flyteworkflows.flyte.lyft.com|flyteworkflows.flyte.lyft.com>)
And
Copy code
unable to fire the schedule {BaseModel:{ID:1754 CreatedAt:2024-05-15 05:47:49.237057 +0000 UTC UpdatedAt:2024-05-15 05:47:49.237057 +0000 UTC DeletedAt:<nil>} SchedulableEntityKey:{Project:redacted Domain:development Name:redacted_launchplan Version:d0f89907dc495858c49c882cb05b5d564ba859ee} CronExpression:0 12 20 * * FixedRateValue:0 Unit:MINUTE KickoffTimeInputArg: Active:0xc0007655c4} at 2024-06-20 12:00:00 +0000 UTC time due to rpc error: code = Internal desc = failed to create workflow in propeller the server could not find the requested resource (post <http://flyteworkflows.flyte.lyft.com|flyteworkflows.flyte.lyft.com>)
After a while it stops with
Copy code
failed to catch up on all the schedules. Aborting
Now, the weird thing is that this particular launchplan is scheduled to run once a month on the 20th, yet we see it (randomly) attempt to run every couple of days. Where could this come from? We run a control plane / data plane setup where the two run in different aws accounts. That data planes only run when needed, so for example the data plane for this launchplan would not be up on days other than the 20th. (@faint-activity-87590 💗 ) # Answer The issue you are encountering appears to be related to the FlyteScheduler's handling of scheduled tasks, particularly with catching up on missed schedules and attempting to execute them even if the data plane is not available. This behavior is consistent with known issues in Flyte's scheduler. 1. Failed to Create Workflow: The error messages indicate that the scheduler is attempting to create workflows but is failing due to the data plane being unavailable. This is likely because the data plane is not running on days other than the 20th, causing the scheduler to fail when it tries to catch up on missed schedules. 2. Unexpected Execution Attempts: The scheduler might be attempting to catch up on missed schedules due to redeployments or restarts, leading to unexpected execution attempts. This behavior has been observed in Flyte and is documented in this issue, where the scheduler fired launch plans unexpectedly on redeployment.
>>3. Scheduler Catch-Up Mechanism: The scheduler's catch-up mechanism might be causing it to attempt to run the launch plan outside its intended schedule. This is further corroborated by the error message 
failed to catch up on all the schedules. Aborting
, which indicates that the scheduler is trying to catch up on missed schedules but is failing due to the data plane being unavailable.
To mitigate this issue, consider the following steps: - Ensure Data Plane Availability: Ensure that the data plane is available when the scheduler attempts to catch up on missed schedules. This might involve keeping the data plane running or scheduling it to start before the scheduler's catch-up attempts. - Review Scheduler Configuration: Review the scheduler configuration to ensure it aligns with your deployment strategy. You might need to adjust the scheduler's behavior to avoid unnecessary catch-up attempts. Would you like more detailed steps on how to adjust the scheduler configuration or any other specific aspect? Sources: - FlyteScheduler fired launchplans on redeployment unexpectedly - Native scheduler failed silently and restarted when running malformed Launch Plan Slack Link https://flyte-org.slack.com/archives/CP2HDHKE1/p1719567874.389049 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
Open in Slack
PreviousNext
Powered by