late-pencil-3873
03/31/2022, 6:18 PM* * * * *
cron expression via flyteremote.
After that, the flyte UI still shows This workflow has no schedules
, as expected.
I then execute flytectl update launchplan -p default -d development --version fabio-cron-2a3fb7f-dirty-BQN44 cron_hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0 --activate
which gives:
updated launchplan successfully on cron_hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0%
In the UI, the schedule is updated as well, it now says Every minute
.
Problem: I’m not seeing any executions.
The flyte admin shows this in the logs after I activated the launchplan:
{"json":{"src":"launch_plan.go:108"},"level":"info","msg":"Adding resource type for unspecified value in request: [%+v]id:\u003cproject:\"default\" domain:\"development\" name:\"cron_hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0\" version:\"fabio-cron-2a3fb7f-dirty-BQN44\" \u003e state:ACTIVE ","ts":"2022-03-31T18:13:01Z"}
{"json":{"src":"log.go:42"},"level":"warning","msg":"Failed to parse authenticated client metadata when creating audit log","ts":"2022-03-31T18:13:01Z"}
{"json":{"src":"log.go:91"},"level":"info","msg":"Recording request: [{\"Principal\":{\"Subject\":\"\",\"ClientID\":\"\",\"TokenIssuedAt\":\"0001-01-01T00:00:00Z\"},\"Client\":{\"ClientIP\":\"\"},\"Request\":{\"Method\":\"UpdateLaunchPlan\",\"Parameters\":{\"domain\":\"development\",\"name\":\"cron_hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0\",\"project\":\"default\",\"version\":\"fabio-cron-2a3fb7f-dirty-BQN44\"},\"Mode\":1,\"ReceivedAt\":\"2022-03-31T18:13:01.578706629Z\"},\"Response\":{\"ResponseCode\":\"OK\",\"SentAt\":\"2022-03-31T18:13:01.597983728Z\"}}]","ts":"2022-03-31T18:13:01Z"}
The flytescheduler
doesn’t show any update in the logs.
Does anyone have some guidance where to start debugging this?
Thanks!late-pencil-3873
03/31/2022, 6:28 PM# --
# Flyte uses a cloud hosted Cron scheduler to run workflows on a schedule. The following module is optional. Without,
# this module, you will not have scheduled launchplans/workflows.
workflow_scheduler:
enabled: true
type: native
freezing-airport-6809
acceptable-policeman-57188
late-pencil-3873
04/01/2022, 7:46 AMtime="2022-03-31T18:01:37Z" level=info msg="Using config file: [/etc/flyte/config/admin.yaml /etc/flyte/config/db.yaml /etc/flyte/config/logger.yaml /etc/flyte/config/server.yaml]"
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [externalevents] updated. No update handler registered.","ts":"2022-03-31T18:01:38Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [cluster_resources] updated. No update handler registered.","ts":"2022-03-31T18:01:38Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [queues] updated. No update handler registered.","ts":"2022-03-31T18:01:38Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-03-31T18:01:38Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [remotedata] updated. No update handler registered.","ts":"2022-03-31T18:01:38Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [registration] updated. No update handler registered.","ts":"2022-03-31T18:01:38Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [task_resources] updated. No update handler registered.","ts":"2022-03-31T18:01:38Z"}
[33m[2022-03-31 18:01:38][0m [35m[info] registering callback `validations:validate` from /go/pkg/mod/github.com/qor/validations@v0.0.0-20171228122639-f364bca61b46/callbacks.go:74[0m [31;1m [0m
[33m[2022-03-31 18:01:38][0m [35m[info] registering callback `validations:validate` from /go/pkg/mod/github.com/qor/validations@v0.0.0-20171228122639-f364bca61b46/callbacks.go:77[0m [31;1m [0m
{"json":{"src":"client.go:181"},"level":"error","msg":"failed to initialize token source provider. Err: failed to fetch auth metadata. Error: rpc error: code = Unimplemented desc = unknown service flyteidl.service.AuthMetadataService","ts":"2022-03-31T18:01:38Z"}
{"json":{"src":"client.go:186"},"level":"warning","msg":"Starting an unauthenticated client because: can't create authenticated channel without a TokenSourceProvider","ts":"2022-03-31T18:01:38Z"}
{"json":{"src":"client.go:65"},"level":"info","msg":"Initialized Admin client","ts":"2022-03-31T18:01:38Z"}
{"json":{"src":"scheduler.go:60"},"level":"info","msg":"Successfully initialized a native flyte scheduler","ts":"2022-03-31T18:01:38Z"}
{"json":{"src":"schedule_executor.go:37"},"level":"info","msg":"Flyte native scheduler started successfully","ts":"2022-03-31T18:01:38Z"}
{"json":{"src":"server.go:96"},"level":"info","msg":"Starting profiling server on port [10254]","ts":"2022-03-31T18:01:38Z"}
{"json":{"src":"schedule_executor.go:57"},"level":"info","msg":"Number of schedules retrieved 0","ts":"2022-03-31T18:01:38Z"}
late-pencil-3873
04/01/2022, 7:47 AMicy-agent-73298
04/01/2022, 8:11 AMflytectl version
late-pencil-3873
04/01/2022, 8:22 AMflytectl version
A new release of flytectl is available: 0.5.8 → v0.5.9
To upgrade, run: brew update && brew upgrade flytectl
<https://github.com/flyteorg/flytectl/releases/tag/v0.5.9>
{
"App": "flytectl",
"Build": "ebda958",
"Version": "0.5.8",
"BuildTime": "2022-04-01 10:22:09.671428 +0200 CEST m=+0.080706115"
}%
icy-agent-73298
04/01/2022, 8:23 AMlate-pencil-3873
04/01/2022, 8:25 AMlate-pencil-3873
04/01/2022, 8:26 AMContainers:
flyteadmin:
Container ID: <containerd://f8538b021cabdbf6d3475f20991e259f3bc68ea5c6903c09197a3358dbb1bc3>7
Image: <http://cr.flyte.org/flyteorg/flyteadmin-release:v0.18.2|cr.flyte.org/flyteorg/flyteadmin-release:v0.18.2>
icy-agent-73298
04/01/2022, 8:28 AMlate-pencil-3873
04/01/2022, 8:28 AMicy-agent-73298
04/01/2022, 8:57 AMflytectl get launchplans -p flytesnacks -d development cron_hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0 -o yaml
I tried using same schedule ( * * * * *
) on sample launch plan that we have in our flytesnacks examples here https://github.com/flyteorg/flytesnacks/blob/f7af27e23b3935a166645cf96a68583cdd263a87/cookbook/core/scheduled_workflows/lp_schedules.py#L55 but that worked well for me
I deployed the latest sandbox though. One thing i could suggest at this point is to upgrade to latest released flyteadmin. The one you are using is 4 months old
https://github.com/flyteorg/flyteadmin/pkgs/container/flyteadmin-release/16570755?tag=v0.19.3icy-agent-73298
04/01/2022, 8:57 AMicy-agent-73298
04/01/2022, 10:11 AMcron_lp = LaunchPlan.get_or_create(
name="my_cron_scheduled_lp",
workflow=date_formatter_wf,
schedule=CronSchedule(
# Note that kickoff_time_input_arg matches the workflow input we defined above: kickoff_time
# But in case you are using the AWS scheme of schedules and not using the native scheduler then switch over the schedule parameter with cron_expression
schedule="* * * * *", # Following schedule runs every min
kickoff_time_input_arg="kickoff_time",
),
)
icy-agent-73298
04/01/2022, 10:14 AM2022-03-31T18:12:32Z
First deactivation sent at : 2022-03-31T18:19:51Z
icy-agent-73298
04/01/2022, 10:24 AMflytectl get launchplans -p flytesnacks -d development cron_hydra_workflow_cfg_flyte_workflows.collaborative_filtering -o yaml
would be helpfullate-pencil-3873
04/01/2022, 11:32 AMflytectl update launchplan -p default -d development --version fabio-cron-2a3fb7f-dirty-BQN44 cron_hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0 --activate
gives:
updated launchplan successfully on cron_hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0
Then:
flytectl get launchplans -p default -d development cron_hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0 -o yaml
(Note the different project and workflow name compared to the command you posted)
One thing that might be relevant that I just realized:
I’m accessing flyte via kubectl -n flyte port-forward svc/flyteadmin 30081:81
late-pencil-3873
04/01/2022, 11:36 AMOne additional thing i would like to check is do you have KickOffTimeInputArg on your schedule
eg : cron schedule from flytesnacks with kickofftimeNo, I don’t use this arg, is it required?
icy-agent-73298
04/01/2022, 12:39 PMicy-agent-73298
04/01/2022, 12:41 PMlate-pencil-3873
04/01/2022, 1:28 PMacceptable-policeman-57188
late-pencil-3873
04/01/2022, 7:04 PMflytectl update … --archive
it stopped showing that but still runs the workflow every minute.
flytectl update launchplan -p default -d development --version fabio-cron-2a3fb7f-dirty-3M0U1 hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0 --archive
gives
updated launchplan successfully on hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0
But the command doesn’t cause any logs in the flyteschedulerfreezing-airport-6809
freezing-airport-6809
late-pencil-3873
04/01/2022, 7:08 PMacceptable-policeman-57188
acceptable-policeman-57188
{"json":{"lp":"hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0","src":"event_scheduler_impl.go:81"},"level":"info","msg":"Deactivated the schedule {{UNSPECIFIED default development hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0 fabio-cron-2a3fb7f-dirty-3M0U1 {} [] 0} } in the scheduler","ts":"2022-04-01T18:51:27Z"}
{"json":{"lp":"hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0","src":"launch_plan_manager.go:265"},"level":"debug","msg":"disabled launch plan: [resource_type:LAUNCH_PLAN project:\"default\" domain:\"development\" name:\"hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0\" version:\"fabio-cron-2a3fb7f-dirty-3M0U1\" ]","ts":"2022-04-01T18:51:27Z"}
this looks right, why is scheduler still running it?late-pencil-3873
04/01/2022, 7:10 PMlate-pencil-3873
04/01/2022, 7:10 PMlate-pencil-3873
04/01/2022, 7:11 PMflytectl -p default -d development get launchplan | grep 3M0U1
| fabio-cron-2a3fb7f-dirty-3M0U1 | hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0 | LAUNCH_PLAN | 2022-04-01T18:36:28.931942Z |
late-pencil-3873
04/01/2022, 7:12 PMlate-pencil-3873
04/01/2022, 7:12 PMpyflyte lp --help
suggests that there only is a way to activate-all.acceptable-policeman-57188
late-pencil-3873
04/01/2022, 7:19 PM{"json":{"src":"identifier.go:54"},"level":"debug","msg":"Returning hash for [{ResourceType:UNSPECIFIED Project:default Domain:development Name:cron_hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0 Version:fabio-cron-2a3fb7f-dirty-ERM5F XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]: 5967604739973439342","ts":"2022-04-01T19:18:22Z"}
{"json":{"src":"gocron_scheduler.go:148"},"level":"debug","msg":"Job doesn't exists in the map for name 5967604739973439342 with schedule {BaseModel:{ID:1 CreatedAt:2022-04-01 13:42:12.120162 +0000 +0000 UpdatedAt:2022-04-01 13:45:18.437093 +0000 +0000 DeletedAt:\u003cnil\u003e} SchedulableEntityKey:{Project:default Domain:development Name:cron_hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0 Version:fabio-cron-2a3fb7f-dirty-ERM5F} CronExpression:* * * * * FixedRateValue:0 Unit:MINUTE KickoffTimeInputArg:kickoff_time Active:0xc0001f24f8} and hence already removed","ts":"2022-04-01T19:18:22Z"}
{"json":{"src":"identifier.go:54"},"level":"debug","msg":"Returning hash for [{ResourceType:UNSPECIFIED Project:default Domain:development Name:hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0 Version:fabio-cron-2a3fb7f-dirty-BYUKX XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]: 13364209728234539433","ts":"2022-04-01T19:18:22Z"}
{"json":{"src":"gocron_scheduler.go:148"},"level":"debug","msg":"Job doesn't exists in the map for name 13364209728234539433 with schedule {BaseModel:{ID:2 CreatedAt:2022-04-01 18:32:29.471284 +0000 +0000 UpdatedAt:2022-04-01 18:32:47.517206 +0000 +0000 DeletedAt:\u003cnil\u003e} SchedulableEntityKey:{Project:default Domain:development Name:hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0 Version:fabio-cron-2a3fb7f-dirty-BYUKX} CronExpression:* * * * * FixedRateValue:0 Unit:MINUTE KickoffTimeInputArg:kickoff_time Active:0xc0001f26a8} and hence already removed","ts":"2022-04-01T19:18:22Z"}
{"json":{"src":"identifier.go:54"},"level":"debug","msg":"Returning hash for [{ResourceType:UNSPECIFIED Project:default Domain:development Name:hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0 Version:fabio-cron-2a3fb7f-dirty-3M0U1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]: 1598734420952108938","ts":"2022-04-01T19:18:22Z"}
{"json":{"src":"gocron_scheduler.go:148"},"level":"debug","msg":"Job doesn't exists in the map for name 1598734420952108938 with schedule {BaseModel:{ID:3 CreatedAt:2022-04-01 18:37:01.571398 +0000 +0000 UpdatedAt:2022-04-01 19:02:16.956318 +0000 +0000 DeletedAt:\u003cnil\u003e} SchedulableEntityKey:{Project:default Domain:development Name:hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0 Version:fabio-cron-2a3fb7f-dirty-3M0U1} CronExpression: FixedRateValue:1 Unit:MINUTE KickoffTimeInputArg: Active:0xc0001f27e8} and hence already removed","ts":"2022-04-01T19:18:22Z"}
{"json":{"src":"identifier.go:54"},"level":"debug","msg":"Returning hash for [{ResourceType:UNSPECIFIED Project:default Domain:development Name:cron_hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0 Version:fabio-cron-2a3fb7f-dirty-ERM5F XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]: 5967604739973439342","ts":"2022-04-01T19:18:52Z"}
{"json":{"src":"gocron_scheduler.go:148"},"level":"debug","msg":"Job doesn't exists in the map for name 5967604739973439342 with schedule {BaseModel:{ID:1 CreatedAt:2022-04-01 13:42:12.120162 +0000 +0000 UpdatedAt:2022-04-01 13:45:18.437093 +0000 +0000 DeletedAt:\u003cnil\u003e} SchedulableEntityKey:{Project:default Domain:development Name:cron_hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0 Version:fabio-cron-2a3fb7f-dirty-ERM5F} CronExpression:* * * * * FixedRateValue:0 Unit:MINUTE KickoffTimeInputArg:kickoff_time Active:0xc00077a938} and hence already removed","ts":"2022-04-01T19:18:52Z"}
{"json":{"src":"identifier.go:54"},"level":"debug","msg":"Returning hash for [{ResourceType:UNSPECIFIED Project:default Domain:development Name:hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0 Version:fabio-cron-2a3fb7f-dirty-BYUKX XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]: 13364209728234539433","ts":"2022-04-01T19:18:52Z"}
{"json":{"src":"gocron_scheduler.go:148"},"level":"debug","msg":"Job doesn't exists in the map for name 13364209728234539433 with schedule {BaseModel:{ID:2 CreatedAt:2022-04-01 18:32:29.471284 +0000 +0000 UpdatedAt:2022-04-01 18:32:47.517206 +0000 +0000 DeletedAt:\u003cnil\u003e} SchedulableEntityKey:{Project:default Domain:development Name:hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0 Version:fabio-cron-2a3fb7f-dirty-BYUKX} CronExpression:* * * * * FixedRateValue:0 Unit:MINUTE KickoffTimeInputArg:kickoff_time Active:0xc00077aa88} and hence already removed","ts":"2022-04-01T19:18:52Z"}
{"json":{"src":"identifier.go:54"},"level":"debug","msg":"Returning hash for [{ResourceType:UNSPECIFIED Project:default Domain:development Name:hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0 Version:fabio-cron-2a3fb7f-dirty-3M0U1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]: 1598734420952108938","ts":"2022-04-01T19:18:53Z"}
{"json":{"src":"gocron_scheduler.go:148"},"level":"debug","msg":"Job doesn't exists in the map for name 1598734420952108938 with schedule {BaseModel:{ID:3 CreatedAt:2022-04-01 18:37:01.571398 +0000 +0000 UpdatedAt:2022-04-01 19:02:16.956318 +0000 +0000 DeletedAt:\u003cnil\u003e} SchedulableEntityKey:{Project:default Domain:development Name:hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0 Version:fabio-cron-2a3fb7f-dirty-3M0U1} CronExpression: FixedRateValue:1 Unit:MINUTE KickoffTimeInputArg: Active:0xc00077ab58} and hence already removed","ts":"2022-04-01T19:18:53Z"}
acceptable-policeman-57188
acceptable-policeman-57188
late-pencil-3873
04/01/2022, 7:24 PMlate-pencil-3873
04/01/2022, 7:26 PMgrpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = "failed to create workflow in propeller the server could not find the requested resource (post flyteworkflows.flyte.lyft.com)"
debug_error_string = "{"created":"@1648841090.727274000","description":"Error received from peer ipv4:127.0.0.1:30081","file":"src/core/lib/surface/call.cc","file_line":904,"grpc_message":"failed to create workflow in propeller the server could not find the requested resource (post flyteworkflows.flyte.lyft.com)","grpc_status":13}"
>
Maybe I bricked something. I think it might be best if I clear the database and the buckets and start fresh, see if the problem persists. Luckily it is not a system where I cannot do that 🙂acceptable-policeman-57188
late-pencil-3873
04/01/2022, 7:29 PMlate-pencil-3873
04/01/2022, 7:31 PMlate-pencil-3873
04/01/2022, 7:31 PMlate-pencil-3873
04/01/2022, 7:31 PMlate-pencil-3873
04/01/2022, 7:31 PMacceptable-policeman-57188
kubectl get fly
and see if that even returns successfullylate-pencil-3873
04/01/2022, 7:32 PMlate-pencil-3873
04/01/2022, 7:32 PMacceptable-policeman-57188
acceptable-policeman-57188
late-pencil-3873
04/01/2022, 7:33 PMah okay starting fresh should help 🙂 hope you have a good weekend too, i think i found an issue with the CronSchedule in the scheduler but I’ll follow up with Prafulla offline to verify and/or fixI was using the FixedRateScheduler now.
icy-agent-73298
04/04/2022, 6:57 AMlate-pencil-3873
04/04/2022, 7:43 AM