little-cricket-84530
04/26/2023, 6:38 PM{
"id": {
"resourceType": "LAUNCH_PLAN",
"project": "my_project",
"domain": "production",
"name": "my_lp",
"version": "20949aa7bbea5421dfef9f96d4248c7a26a6f5d5"
},
"spec": {
"workflowId": {
"resourceType": "WORKFLOW",
"project": "my_project",
"domain": "production",
"name": "some_wf",
"version": "20949aa7bbea5421dfef9f96d4248c7a26a6f5d5"
},
"entityMetadata": {
"schedule": {
"rate": {
"value": 8,
"unit": "HOUR"
}
}
},
"defaultInputs": {},
"fixedInputs": {},
"labels": {},
"annotations": {},
"rawOutputDataConfig": {}
},
"closure": {
"state": "ACTIVE",
"expectedInputs": {},
"expectedOutputs": {},
"createdAt": "2023-04-26T18:23:39.503710Z",
"updatedAt": "2023-04-26T18:23:39.503710Z"
}
}
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
icy-agent-73298
04/27/2023, 4:35 AMlittle-cricket-84530
04/27/2023, 4:54 AMlittle-cricket-84530
04/27/2023, 4:59 AMflytectl update launchplan --admin.endpoint $(FLYTE_ADMIN_HOST):443 -p $(FLYTE_PROJECT_NAME) -d $(FLYTE_PROJECT_DOMAIN) \
$(name) --activate --version $(IMAGE_VERSION)"
little-cricket-84530
04/27/2023, 5:12 AM---------------------- ------------------- ------------------------------------------ ------------- ----------- ---------------------- -------------------------------- ----------------- -------------------- --------------------
| NAME | LAUNCH PLAN NAME | VERSION | TYPE | PHASE | SCHEDULED TIME | STARTED | ELAPSED TIME | ABORT DATA (TRUNC) | ERROR DATA (TRUNC) |
---------------------- ------------------- ------------------------------------------ ------------- ----------- ---------------------- -------------------------------- ----------------- -------------------- --------------------
| f9618b52b437ae85d000 | my_pipeline_lp | b699495922ea41c0849e7bd03f190783475552d3 | LAUNCH_PLAN | SUCCEEDED | 2023-04-26T18:21:44Z | 2023-04-26T18:21:49.735450044Z | 3565.693426035s | | |
---------------------- ------------------- ------------------------------------------ ------------- ----------- ---------------------- -------------------------------- ----------------- -------------------- --------------------
---------------------- ------------------- ------------------------------------------ ------------- --------- ---------------------- -------------------------------- ---------------- -------------------- --------------------
| NAME | LAUNCH PLAN NAME | VERSION | TYPE | PHASE | SCHEDULED TIME | STARTED | ELAPSED TIME | ABORT DATA (TRUNC) | ERROR DATA (TRUNC) |
---------------------- ------------------- ------------------------------------------ ------------- --------- ---------------------- -------------------------------- ---------------- -------------------- --------------------
| f9318b52b437ae85d000 | my_pipeline_lp | b699495922ea41c0849e7bd03f190783475552d3 | LAUNCH_PLAN | ABORTED | 2023-04-26T182141Z | 2023-04-26T182147.419096843Z | 170.893382172s | Terminated from UI | |
---------------------- ------------------- ------------------------------------------ ------------- --------- ---------------------- -------------------------------- ---------------- -------------------- --------------------little-cricket-84530
04/27/2023, 5:13 AMicy-agent-73298
04/27/2023, 5:29 AMlittle-cricket-84530
04/27/2023, 5:12 PMlittle-cricket-84530
04/27/2023, 5:47 PMicy-agent-73298
04/27/2023, 7:05 PMlittle-cricket-84530
04/27/2023, 7:06 PMlittle-cricket-84530
04/27/2023, 7:06 PMlittle-cricket-84530
04/27/2023, 7:08 PMlittle-cricket-84530
04/27/2023, 7:09 PMlittle-cricket-84530
04/27/2023, 7:10 PMicy-agent-73298
04/27/2023, 7:11 PMicy-agent-73298
04/27/2023, 7:11 PMBut the other pipelines with schedules are working perfectlyThats odd that its just affecting this launchplan
little-cricket-84530
04/27/2023, 7:12 PMlittle-cricket-84530
04/27/2023, 7:12 PMlittle-cricket-84530
04/27/2023, 7:14 PMicy-agent-73298
04/27/2023, 7:14 PMflytectl get launchplan -p project -d domain name -o yaml
little-cricket-84530
04/27/2023, 7:15 PMlittle-cricket-84530
04/27/2023, 7:19 PMicy-agent-73298
04/27/2023, 7:33 PMicy-agent-73298
04/27/2023, 8:34 PMlittle-cricket-84530
04/27/2023, 8:36 PMlittle-cricket-84530
04/27/2023, 8:37 PMlittle-cricket-84530
04/27/2023, 8:38 PMlittle-cricket-84530
04/27/2023, 8:45 PMicy-agent-73298
04/28/2023, 1:48 AM2023-04-26T10:21:33.678573Z
icy-agent-73298
04/28/2023, 1:50 AM2023-04-26T18:21:44Z
2023-04-26T18:21:41Z
many-exabyte-15566
04/30/2023, 11:11 PMicy-agent-73298
05/01/2023, 7:11 AMicy-agent-73298
05/01/2023, 7:35 AMlogger.yaml: |
logger:
level: 6
I also wrote a unit to check for the above params like this
func TestGetScheduleName(t *testing.T) {
ctx := context.Background()
schedule := models.SchedulableEntity{
BaseModel: adminModels.BaseModel{
ID: 2,
UpdatedAt: time.Now(),
},
SchedulableEntityKey: models.SchedulableEntityKey{
Project: "navigation",
Domain: "production",
Name: "daily_pipeline_8hr_lp",
Version: "b699495922ea41c0849e7bd03f190783475552d3",
},
Unit: admin.FixedRateUnit_HOUR * 8,
KickoffTimeInputArg: "kickoff_time",
}
str := GetScheduleName(ctx, schedule)
assert.Equal(t, "17193008309657693892", str)
}
And the GetScheduleName is only dependent on these
Project: "navigation",
Domain: "production",
Name: "daily_pipeline_8hr_lp",
Version: "b699495922ea41c0849e7bd03f190783475552d3",
https://github.com/flyteorg/flyteadmin/blob/master/scheduler/identifier/identifier.go#L26
From the previous conversations for these execution ID’s f9618b52b437ae85d000 and f9318b52b437ae85d000 the above launchplan params needed to generate schedule name are the same so this line should tell if another has been added https://github.com/flyteorg/flyteadmin/blob/master/scheduler/core/gocron_scheduler.go#L117
and add the schedule at this line https://github.com/flyteorg/flyteadmin/blob/master/scheduler/core/gocron_scheduler.go#L139
But for some wierd reason we are seeing L139 being executed twice for the same launchplan params . Suspecting some race if thats the case but still would be great to pin point from the logs this exact issue. Having the scheduler logs at debug level would help loticy-agent-73298
05/01/2023, 4:35 PMmany-exabyte-15566
05/01/2023, 4:54 PMmany-exabyte-15566
05/01/2023, 4:54 PMicy-agent-73298
05/01/2023, 5:17 PMmany-exabyte-15566
05/01/2023, 6:34 PMmany-exabyte-15566
05/01/2023, 6:35 PM