We are seeing many messages such as this one right after deployment:

failed to create execution create request %+v due to %vproject:"redacted" domain:"development" name:"ffc074c35db9fa02c000" spec:<launch_plan:<resource_type:LAUNCH_PLAN project:"redacted" domain:"development" name:"redacted_launchplan" version:"d0f89907dc495858c49c882cb05b5d564ba859ee" > metadata:<mode:SCHEDULED scheduled_at:<seconds:1718884800 > > > inputs:<>  rpc error: code = Internal desc = failed to create workflow in propeller the server could not find the requested resource (post <http://flyteworkflows.flyte.lyft.com|flyteworkflows.flyte.lyft.com>)

And

unable to fire the schedule {BaseModel:{ID:1754 CreatedAt:2024-05-15 05:47:49.237057 +0000 UTC UpdatedAt:2024-05-15 05:47:49.237057 +0000 UTC DeletedAt:<nil>} SchedulableEntityKey:{Project:redacted Domain:development Name:redacted_launchplan Version:d0f89907dc495858c49c882cb05b5d564ba859ee} CronExpression:0 12 20 * * FixedRateValue:0 Unit:MINUTE KickoffTimeInputArg: Active:0xc0007655c4} at 2024-06-20 12:00:00 +0000 UTC time due to rpc error: code = Internal desc = failed to create workflow in propeller the server could not find the requested resource (post <http://flyteworkflows.flyte.lyft.com|flyteworkflows.flyte.lyft.com>)

After a while it stops with

failed to catch up on all the schedules. Aborting

Now, the weird thing is that this particular launchplan is scheduled to run once a month on the 20th, yet we see it (randomly) attempt to run every couple of days. Where could this come from? We run a control plane / data plane setup where the two run in different aws accounts. That data planes only run when needed, so for example the data plane for this launchplan would not be up on days other than the 20th. (@faint-activity-87590 💗 )

Hey @microscopic-school-54375 good to see you again 🙂 Any recent change? From the error it's like the CRD is not in the cluster

But I guess the question is why are we firing the scheduler

Scheduler tries to catch up any schedules that it missed running. If you dont want those to be running then you can deactivate the schedule https://docs.flyte.org/en/latest/user_guide/productionizing/schedules.html#deactivating-a-schedule Seems since the scheduler is failing to run that schedule, every time you start it will attempt to run that schedule again until it completes successfully or you can deactivate the schedule