Jing Li
03/23/2022, 7:38 AMJing Li
03/23/2022, 7:39 AMJing Li
03/23/2022, 7:41 AMEduardo Apolinario (eapolinario)
03/23/2022, 2:00 PMSandra Youssef
03/23/2022, 11:21 PMcommunity updates▾
https://youtu.be/9dkd0rxy4bl ▾
Ketan (kumare3)
03/24/2022, 3:51 AMSamhita Alla
03/24/2022, 3:29 PMKetan (kumare3)
03/24/2022, 6:40 PMJing Li
03/25/2022, 9:35 AMHampus Rosvall
03/25/2022, 10:53 AMFlyteAdmin
? It is in our road map to bump the ALB Ingress Controller version, but until that is done I would still like to experiment with Flyte on EKS.jeev
03/25/2022, 6:56 PMHafsa Junaid
03/25/2022, 7:04 PMHafsa Junaid
03/26/2022, 12:38 AMhelm install incubator/sparkoperator --namespace spark-operator --kubeconfig=~/.flyte/k3s/k3s.yaml
Error:
Error: INSTALLATION FAILED: must either provide a name or specify --generate-name
What is generate name here?Alex Bain
03/26/2022, 1:08 AMv0.19.3
. No matter what I add to my Propeller deployment.yaml
, I cannot seem to get Stackdriver log links to render in Flyte Console for map tasks (i.e. k8s_array
tasks). Map task subtasks always render with Kubernetes log links (even when declaring kubernetes-enabled: false
). I'm pretty sure that Stackdriver log links just don't work yet for map tasks. Could you confirm that?Hafsa Junaid
03/26/2022, 2:55 PMAlex Pozimenko
03/28/2022, 7:32 PMflyte-propeller-all-workflow-acceptance.latency.ms.sum
? Sample chat below. Or perhaps I should be looking at percentiles only (.ms.0.9 and .ms.0.99)?Fernando Diaz
03/29/2022, 10:17 AMbigqueryjob
). We see in the logs that the node timeouts as expected:
"Current execution for the node timed out; timeout configured: 3h0m0s"
However, the Delete
method of the plugin (https://github.com/flyteorg/flyteplugins/blob/master/go/tasks/plugins/webapi/bigquery/plugin.go#L226) is never called after the timeout, so the job is not cancelled. Is this expected? How can I tell propeller to abort the node after a timeout?Hampus Rosvall
03/29/2022, 12:33 PM(.venv) ☁ ~ ⚡ make register PROJECT=determined DOMAIN=development
flytectl register files \
--project determined \
--domain development \
--archive flyte-package.tgz \
--version v1-fast1
Error: please check your Storage Config. It failed while uploading the source code. Failed to write data [69b] to path [fast/v1-fast1-fast9bb248f6c3ae89386ec7105633eaaee8.tar.gz].: PutObject, putting object: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
gmake: *** [Makefile:22: register] Error 1
(.venv) ☁ ~ cat ~/.flyte/config.yaml
admin:
endpoint: dns:///localhost:8089
authType: Pkce
insecure: true
logger:
show-source: true
level: 0
storage:
type: stow
stow:
kind: s3
config:
auth_type: iam
region: eu-west-1
container: <name-of-my-bucket>
Mohamed Nabil H
03/29/2022, 5:02 PMMohamed Nabil H
03/29/2022, 5:06 PMMatheus Moreno
03/29/2022, 5:20 PMMiggy
03/29/2022, 10:00 PMMike Ossareh
03/30/2022, 5:13 PMAlex Pozimenko
03/30/2022, 5:49 PMflyte-admin-execution.manager-active.executions.gauge
shows negative values. Is this by design?Sandra Youssef
03/30/2022, 7:38 PMMiggy
03/30/2022, 8:39 PMterminated in the background, manually
. I took a look at the code here and see that this occurs when the pod get deleted. I don’t see a delete request being sent to admin so don’t think there is anything there. Propeller logs haven’t shown anything either. So I’ve been looking at kube logs to try to see what is triggering the pod deleted but there’s a lot of noise in the logs so taking some time to through them. Meanwhile, I figured I’d ask here if anyone has run into something similar beforeKetan (kumare3)
03/31/2022, 4:02 AMFabio Grätz
03/31/2022, 6:18 PM* * * * *
cron expression via flyteremote.
After that, the flyte UI still shows This workflow has no schedules
, as expected.
I then execute flytectl update launchplan -p default -d development --version fabio-cron-2a3fb7f-dirty-BQN44 cron_hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0 --activate
which gives:
updated launchplan successfully on cron_hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0%
In the UI, the schedule is updated as well, it now says Every minute
.
Problem: I’m not seeing any executions.
The flyte admin shows this in the logs after I activated the launchplan:
{"json":{"src":"launch_plan.go:108"},"level":"info","msg":"Adding resource type for unspecified value in request: [%+v]id:\u003cproject:\"default\" domain:\"development\" name:\"cron_hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0\" version:\"fabio-cron-2a3fb7f-dirty-BQN44\" \u003e state:ACTIVE ","ts":"2022-03-31T18:13:01Z"}
{"json":{"src":"log.go:42"},"level":"warning","msg":"Failed to parse authenticated client metadata when creating audit log","ts":"2022-03-31T18:13:01Z"}
{"json":{"src":"log.go:91"},"level":"info","msg":"Recording request: [{\"Principal\":{\"Subject\":\"\",\"ClientID\":\"\",\"TokenIssuedAt\":\"0001-01-01T00:00:00Z\"},\"Client\":{\"ClientIP\":\"\"},\"Request\":{\"Method\":\"UpdateLaunchPlan\",\"Parameters\":{\"domain\":\"development\",\"name\":\"cron_hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0\",\"project\":\"default\",\"version\":\"fabio-cron-2a3fb7f-dirty-BQN44\"},\"Mode\":1,\"ReceivedAt\":\"2022-03-31T18:13:01.578706629Z\"},\"Response\":{\"ResponseCode\":\"OK\",\"SentAt\":\"2022-03-31T18:13:01.597983728Z\"}}]","ts":"2022-03-31T18:13:01Z"}
The flytescheduler
doesn’t show any update in the logs.
Does anyone have some guidance where to start debugging this?
Thanks!Alex Bain
03/31/2022, 7:58 PMWilliam Young
03/31/2022, 9:18 PMWilliam Young
03/31/2022, 9:18 PMKetan (kumare3)
03/31/2022, 9:34 PMWilliam Young
03/31/2022, 9:38 PMKetan (kumare3)
03/31/2022, 11:04 PMWilliam Young
04/01/2022, 1:19 PMKetan (kumare3)
04/01/2022, 1:24 PMWilliam Young
04/01/2022, 3:19 PMKetan (kumare3)
04/01/2022, 3:21 PMWilliam Young
04/06/2022, 3:12 PMKetan (kumare3)
04/06/2022, 5:08 PM