SeungTaeKim
08/18/2022, 9:41 AM2022/08/18 08:08:18 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/execution_repo.go:61
[21.243ms] [rows:1] UPDATE "executions" SET "id"=358,"created_at"='2022-08-18 08:05:44.14',"updated_at"='2022-08-18 08:08:18.501',"execution_project"='flytesnacks',"execution_domain"='development',"execution_name"='f985b6eddf4c43d5f000',"launch_plan_id"=114,"workflow_id"=106,"phase"='SUCCEEDED',"closure"='<binary>',"spec"='<binary>',"started_at"='2022-08-18 08:05:49.251',"execution_created_at"='2022-08-18 08:05:44.139',"execution_updated_at"='2022-08-18 08:08:18.497',"duration"='2m29.245844975s',"mode"=1,"inputs_uri"='<s3://my-s3-bucket/metadata/flytesnacks/development/f985b6eddf4c43d5f000/inputs>',"user_inputs_uri"='<s3://my-s3-bucket/metadata/flytesnacks/development/f985b6eddf4c43d5f000/user_inputs>',"user"='a57ebf23-a6e8-45e7-bcb1-52b69f508f67',"state"=0 WHERE "execution_project" = 'flytesnacks' AND "execution_domain" = 'development' AND "execution_name" = 'f985b6eddf4c43d5f000'
{"json":{"exec_id":"f985b6eddf4c43d5f000","src":"execution_manager.go:1588"},"level":"debug","msg":"publishing notifications for execution [project:\"flytesnacks\" domain:\"development\" name:\"f985b6eddf4c43d5f000\" ] in state [SUCCEEDED] for notifications [[phases:SUCCEEDED email:\u003crecipients_email:\"<my email>\" \u003e ]]","ts":"2022-08-18T08:08:18Z"}
{"json":{"exec_id":"f985b6eddf4c43d5f000","src":"publisher.go:30"},"level":"debug","msg":"Publishing the following message [recipients_email:\"<my mail>\" sender_email:\"<sender mail setup in SES>\" subject_line:\"Notice: Execution \\\"flyte_fixedrate.positive_wf\\\" has succeeded in \\\"development\\\".\" body:\"Execution \\\\\\\"flyte_fixedrate.positive_wf [f985b6eddf4c43d5f000]\\\\\\\" has succeeded in \\\\\\\"development\\\\\\\". View details at \u003ca href=\\\\<http://flow.qraftpilot.com/console/projects/flytesnacks/domains/development/executions/f985b6eddf4c43d5f000>\u003e http://<flyte url>/console/projects/flytesnacks/domains/development/executions/f985b6eddf4c43d5f000\u003c/a\u003e. \\n\" ]","ts":"2022-08-18T08:08:18Z"}
{"json":{"exec_id":"faf3d92fb8453e24d000","node":"n0","src":"noop_notifications.go:32"},"level":"debug","msg":"call to noop publish with notification type [flyteidl.admin.NodeExecutionEventRequest] and proto message [event:\u003cid:\u003cnode_id:\"n0\" execution_id:\u003cproject:\"flytesnacks\" domain:\"development\" name:\"faf3d92fb8453e24d000\" \u003e \u003e producer_id:\"propeller\" phase:SUCCEEDED occurred_at:\u003cseconds:1660810338 nanos:222102672 \u003e input_uri:\"<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-faf3d92fb8453e24d000/n0/data/inputs.pb>\" output_uri:\"<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-faf3d92fb8453e24d000/n0/data/0/outputs.pb>\" task_node_metadata:\u003c\u003e spec_node_id:\"n0\" node_name:\"be_positive\" event_version:1 deck_uri:\"<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-faf3d92fb8453e24d000/n0/data/0/deck.html>\" \u003e ]","ts":"2022-08-18T08:12:18Z"}
thank you!Prafulla Mahindrakar
08/18/2022, 9:44 AM{"json":{"exec_id":"faf3d92fb8453e24d000","node":"n0","src":"noop_notifications.go:32"},"level":"debug","msg":"call to noop publish with notification type [flyteidl.admin.NodeExecutionEventRequest]
SeungTaeKim
08/18/2022, 9:50 AMworkflow_notifications:
# enabled: false
enabled: true
config:
notifications:
type: aws
region: "ap-northeast-2"
publisher:
topicName: "arn:aws:sns:ap-northeast-2:<account ID>:<sns name>"
processor:
queueName: "<sqs name>"
accountId: "account id"
emailer:
subject: "Notice: Execution \"{{ workflow.name }}\" has {{ phase }} in \"{{ domain }}\"."
sender: "<ses vertified mail>"
body: >
Execution \"{{ workflow.name }} [{{ name }}]\" has {{ phase }} in \"{{ domain }}\". View details at
<a href=\http://<flyte url>/console/projects/{{ project }}/domains/{{ domain }}/executions/{{ name }}>
http://<flyte url>/console/projects/{{ project }}/domains/{{ domain }}/executions/{{ name }}</a>. {{ error }}
Prafulla Mahindrakar
08/18/2022, 9:53 AMkubectl get configmap -n <flyte-name-space> flyte-admin-base-config -o yaml
SeungTaeKim
08/18/2022, 9:56 AMnotifications.yaml: |
notifications:
emailer:
body: |
Execution \"{{ workflow.name }} [{{ name }}]\" has {{ phase }} in \"{{ domain }}\". View details at
<a href=\http://<flyte url>/console/projects/{{ project }}/domains/{{ domain }}/executions/{{ name }}>
http://<flyte url>/console/projects/{{ project }}/domains/{{ domain }}/executions/{{ name }}</a>. {{ error }}
sender: <ses vertified mail>
subject: 'Notice: Execution "{{ workflow.name }}" has {{ phase }} in "{{ domain
}}".'
processor:
accountId: "account id"
queueName: <sqs name>
publisher:
topicName: arn:aws:sns:ap-northeast-2:<account id>:<sns name>
region: ap-northeast-2
type: aws
Prafulla Mahindrakar
08/18/2022, 10:01 AMUsing default noop notifications processor implementation for config type
Jake Yoon
08/18/2022, 10:35 AM(base) qraft@ai-server0:~/k8s/temp$ k logs -n flyte flyteadmin-cc87d6cf6-dxhn6 | grep noop
{"json":{"src":"factory.go:104"},"level":"info","msg":"Using default noop workflow executor implementation for cloud provider type [local]","ts":"2022-08-18T10:31:14Z"}
{"json":{"exec_id":"f4648f22385a18194000","src":"noop_notifications.go:32"},"level":"debug","msg":"call to noop publish with notification type [flyteidl.admin.WorkflowExecutionEventRequest] and proto message [event:\u003cexecution_id:\u003cproject:\"flytesnacks\" domain:\"development\" name:\"f4648f22385a18194000\" \u003e producer_id:\"propeller\" phase:RUNNING occurred_at:\u003cseconds:1660818728 nanos:781959937 \u003e \u003e ]","ts":"2022-08-18T10:32:08Z"}
Prafulla Mahindrakar
08/18/2022, 10:37 AMSeungTaeKim
08/19/2022, 1:42 AMPrafulla Mahindrakar
08/19/2022, 5:07 AMSeungTaeKim
08/19/2022, 6:19 AMPrafulla Mahindrakar
08/19/2022, 6:21 AMSeungTaeKim
08/19/2022, 6:24 AM> k logs -n flyte flyteadmin-6f5c568d57-2x68w | grep <mailto:seungtae.kim@qraftec.com|seungtae.kim@qraftec.com>
{"json":{"exec_id":"f2ede5902d2074e73000","src":"publisher.go:30"},"level":"debug","msg":"Publishing the following message [recipients_email:\"<my mail>\" sender_email:\"<ses vertified mail>\" subject_line:\"Flyte: 'examples.flyte_fixedrate.positive_wf' has succeeded in 'development'.\" body:\"Execution 'examples.flyte_fixedrate.positive_wf [f2ede5902d2074e73000]' has succeeded in 'development'. View details at\\n\u003ca href=\\\\http://<flyte url>/console/projects/flytesnacks/domains/development/executions/f2ede5902d2074e73000\u003e\\nhttp://<flyte url>/console/projects/flytesnacks/domains/development/executions/f2ede5902d2074e73000\u003c/a\u003e. \\n\" ]","ts":"2022-08-19T06:03:47Z"}
{"json":{"exec_id":"fbde63dfdd160676a000","src":"execution_manager.go:1602"},"level":"debug","msg":"publishing notifications for execution [project:\"flytesnacks\" domain:\"development\" name:\"fbde63dfdd160676a000\" ] in state [SUCCEEDED] for notifications [[phases:SUCCEEDED email:\u003crecipients_email:\"<my mail>\" \u003e ]]","ts":"2022-08-19T06:13:47Z"}
Prafulla Mahindrakar
08/19/2022, 6:26 AMSeungTaeKim
08/19/2022, 6:29 AMError from server (BadRequest): container "flyteadmin" in pod "flyteadmin-6f64bf5bc5-r7m8f" is waiting to start: PodInitializing
2022/08/19 07:04:33 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/task_execution_repo.go:84 SLOW SQL >= 200ms
[832.551ms] [rows:1] UPDATE "task_executions" SET "id"=545,"created_at"='2022-08-19 07:04:23.05',"updated_at"='2022-08-19 07:04:32.845',"deleted_at"=NULL,"phase"='RUNNING',"phase_version"=1,"input_uri"='<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-a8p8cnvbtsw2b4j6bpn2/n0/data/inputs.pb>',"closure"='<binary>',"started_at"='2022-08-19 07:04:32.835',"task_execution_created_at"='2022-08-19 07:04:23.046',"task_execution_updated_at"='2022-08-19 07:04:32.835',"duration"='0s' WHERE "project" = 'flytesnacks' AND "domain" = 'development' AND "name" = 'noti_test.be_positive' AND "version" = 'fgllg7ge0r_5JKIdNVMGHQ==' AND "execution_project" = 'flytesnacks' AND "execution_domain" = 'development' AND "execution_name" = 'a8p8cnvbtsw2b4j6bpn2' AND "node_id" = 'n0' AND "retry_attempt" = 0
{"json":{"exec_id":"a8p8cnvbtsw2b4j6bpn2","node":"n0","src":"noop_notifications.go:32"},"level":"debug","msg":"call to noop publish with notification type [flyteidl.admin.TaskExecutionEventRequest] and proto message [event:\u003ctask_id:\u003cresource_type:TASK project:\"flytesnacks\" domain:\"development\" name:\"noti_test.be_positive\" version:\"fgllg7ge0r_5JKIdNVMGHQ==\" \u003e parent_node_execution_id:\u003cnode_id:\"n0\" execution_id:\u003cproject:\"flytesnacks\" domain:\"development\" name:\"a8p8cnvbtsw2b4j6bpn2\" \u003e \u003e phase:RUNNING producer_id:\"propeller\" logs:\u003curi:\"<https://logs.qraftpilot.com/app/discover#/?_g=(time:(from:now-1w,to:now))>\u0026_a=(columns:!(log),filters:!((query:(match_phrase:(kubernetes.namespace_name:flytesnacks-development))),(query:(match_phrase:(kubernetes.pod_name:a8p8cnvbtsw2b4j6bpn2-n0-0)))),sort:!(!('@timestamp',asc)))\" name:\"Kubernetes Logs (User)\" message_format:JSON \u003e occurred_at:\u003cseconds:1660892672 nanos:835657908 \u003e input_uri:\"<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-a8p8cnvbtsw2b4j6bpn2/n0/data/inputs.pb>\" phase_version:1 task_type:\"python-task\" metadata:\u003cgenerated_name:\"a8p8cnvbtsw2b4j6bpn2-n0-0\" plugin_identifier:\"container\" \u003e event_version:1 \u003e ]","ts":"2022-08-19T07:04:33Z"}
{"json":{"exec_id":"a8p8cnvbtsw2b4j6bpn2","node":"n0","src":"task_execution_manager.go:219"},"level":"debug","msg":"Successfully recorded task execution event [task_id:\u003cresource_type:TASK project:\"flytesnacks\" domain:\"development\" name:\"noti_test.be_positive\" version:\"fgllg7ge0r_5JKIdNVMGHQ==\" \u003e parent_node_execution_id:\u003cnode_id:\"n0\" execution_id:\u003cproject:\"flytesnacks\" domain:\"development\" name:\"a8p8cnvbtsw2b4j6bpn2\" \u003e \u003e phase:RUNNING producer_id:\"propeller\" logs:\u003curi:\"<https://logs.qraftpilot.com/app/discover#/?_g=(time:(from:now-1w,to:now))>\u0026_a=(columns:!(log),filters:!((query:(match_phrase:(kubernetes.namespace_name:flytesnacks-development))),(query:(match_phrase:(kubernetes.pod_name:a8p8cnvbtsw2b4j6bpn2-n0-0)))),sort:!(!('@timestamp',asc)))\" name:\"Kubernetes Logs (User)\" message_format:JSON \u003e occurred_at:\u003cseconds:1660892672 nanos:835657908 \u003e input_uri:\"<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-a8p8cnvbtsw2b4j6bpn2/n0/data/inputs.pb>\" phase_version:1 task_type:\"python-task\" metadata:\u003cgenerated_name:\"a8p8cnvbtsw2b4j6bpn2-n0-0\" plugin_identifier:\"container\" \u003e event_version:1 ]","ts":"2022-08-19T07:04:33Z"}
{"json":{"exec_id":"a8p8cnvbtsw2b4j6bpn2","node":"n0","src":"noop_notifications.go:32"},"level":"debug","msg":"call to noop publish with notification type [flyteidl.admin.TaskExecutionEventRequest] and proto message [event:\u003ctask_id:\u003cresource_type:TASK project:\"flytesnacks\" domain:\"development\" name:\"noti_test.be_positive\" version:\"fgllg7ge0r_5JKIdNVMGHQ==\" \u003e parent_node_execution_id:\u003cnode_id:\"n0\" execution_id:\u003cproject:\"flytesnacks\" domain:\"development\" name:\"a8p8cnvbtsw2b4j6bpn2\" \u003e \u003e phase:RUNNING producer_id:\"propeller\" logs:\u003curi:\"<https://logs.qraftpilot.com/app/discover#/?_g=(time:(from:now-1w,to:now))>\u0026_a=(columns:!(log),filters:!((query:(match_phrase:(kubernetes.namespace_name:flytesnacks-development))),(query:(match_phrase:(kubernetes.pod_name:a8p8cnvbtsw2b4j6bpn2-n0-0)))),sort:!(!('@timestamp',asc)))\" name:\"Kubernetes Logs (User)\" message_format:JSON \u003e occurred_at:\u003cseconds:1660892672 nanos:835657908 \u003e input_uri:\"<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-a8p8cnvbtsw2b4j6bpn2/n0/data/inputs.pb>\" phase_version:1 task_type:\"python-task\" metadata:\u003cgenerated_name:\"a8p8cnvbtsw2b4j6bpn2-n0-0\" plugin_identifier:\"container\" \u003e event_version:1 \u003e ]","ts":"2022-08-19T07:04:33Z"}
{"json":{"src":"handlers.go:237"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2022-08-19T07:04:47Z"}
{"json":{"src":"token.go:83"},"level":"debug","msg":"Could not retrieve bearer token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-08-19T07:04:47Z"}
{"json":{"src":"handlers.go:247"},"level":"info","msg":"Failed to parse Access Token from context. Will attempt to find IDToken. Error: [JWT_VERIFICATION_FAILED] Could not retrieve bearer token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-08-19T07:04:47Z"}
{"json":{"src":"token.go:103"},"level":"debug","msg":"Could not retrieve id token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken","ts":"2022-08-19T07:04:47Z"}
{"json":{"src":"handlers.go:193"},"level":"debug","msg":"gRPC server info in logging interceptor []method [/flyteidl.service.AdminService/CreateTaskEvent]\n","ts":"2022-08-19T07:04:47Z"}
Prafulla Mahindrakar
08/19/2022, 7:21 AMSeungTaeKim
08/19/2022, 7:25 AM# -- Resource manager configuration
resource_manager:
# -- resource manager configuration
propeller:
resourcemanager:
type: noop
should I change this type into aws in values.yaml?Prafulla Mahindrakar
08/19/2022, 7:26 AMSeungTaeKim
08/19/2022, 7:27 AMPrafulla Mahindrakar
08/19/2022, 7:29 AMthis is the configmap settings
```notifications.yaml: |
notifications:
emailer:
body: |
Execution \"{{ workflow.name }} [{{ name }}]\" has {{ phase }} in \"{{ domain }}\". View details at
<a href=\http://<flyte url>/console/projects/{{ project }}/domains/{{ domain }}/executions/{{ name }}>
http://<flyte url>/console/projects/{{ project }}/domains/{{ domain }}/executions/{{ name }}</a>. {{ error }}
sender: <ses vertified mail>
subject: 'Notice: Execution "{{ workflow.name }}" has {{ phase }} in "{{ domain
}}".'
processor:
accountId: "account id"
queueName: <sqs name>
publisher:
topicName: arnawssnsap northeast 2<account id>:<sns name>
region: ap-northeast-2
type: aws```This was the config you shared earlier which looked correct . Can you check you notifications config , the one you shared now is from resourcemanager
SeungTaeKim
08/19/2022, 7:39 AMJake Yoon
08/19/2022, 7:46 AM{
"json": {
"src": "factory.go:104"
},
"level": "info",
"msg": "Using default noop workflow executor implementation for cloud provider type [local]",
"ts": "2022-08-19T07:33:01Z"
}
Prafulla Mahindrakar
08/19/2022, 7:52 AMworkflowExecutor:
scheme: local
local:
You can specify a different scheme for each module. In case of notifications its defined with type field.
Can we debug this through a call since we went back and forth on thisAlso you can grep this for me in the flyteadmin podsAt this point later in our conversation you had the right configUsing default noop notifications processor implementation for config type
I have checked the notification 10 minutes ago, there is aws_processor and it’s working now.
Do I have set additional things up in AWS (SNS or SQS)?
Screenshot from 2022-08-19 10-40-46.png
Jake Yoon
08/22/2022, 8:58 AM{
"json": {
"src": "aws_processor.go:52"
},
"level": "error",
"msg": "failed to unmarshall JSON message [ChhzZXVuZ3RhZS5raW1AcXJhZnRlYy5jb20SEW1sb3BzQHFyYWZ0ZWMuY29tGk1GbHl0ZTogJ2V4YW1wbGVzLmZseXRlX2ZpeGVkcmF0ZS5wb3NpdGl2ZV93ZicgaGFzIHN1Y2NlZWRlZCBpbiAnZGV2ZWxvcG1lbnQnLiLgAkV4ZWN1dGlvbiAnZXhhbXBsZXMuZmx5dGVfZml4ZWRyYXRlLnBvc2l0aXZlX3dmIFtmZWUyMmVjZDM2MDE4NWMzZDAwMF0nIGhhcyBzdWNjZWVkZWQgaW4gJ2RldmVsb3BtZW50Jy4gVmlldyBkZXRhaWxzIGF0CjxhIGhyZWY9XGh0dHA6Ly9mbG93LnFyYWZ0cGlsb3QuY29tL2NvbnNvbGUvcHJvamVjdHMvZmx5dGVzbmFja3MvZG9tYWlucy9kZXZlbG9wbWVudC9leGVjdXRpb25zL2ZlZTIyZWNkMzYwMTg1YzNkMDAwPgpodHRwOi8vZmxvdy5xcmFmdHBpbG90LmNvbS9jb25zb2xlL3Byb2plY3RzL2ZseXRlc25hY2tzL2RvbWFpbnMvZGV2ZWxvcG1lbnQvZXhlY3V0aW9ucy9mZWUyMmVjZDM2MDE4NWMzZDAwMDwvYT4uIAo=] from processor with err: invalid character 'C' looking for beginning of value",
"ts": "2022-08-22T08:48:26Z"
}
Because I saw there was an option in flyteadmin/pkg/async/notifications/factory.go
which is enable64decoding
, I changed it to false but no luck.
Strange thing is the message is entirely encoded in base64, but the flyteadmin
expects it to be JSON formatted at first, and then later it decodes messages using base64.Prafulla Mahindrakar
08/22/2022, 10:11 AMRaw message delivery
on the sqs subscription from sns topicecho "ChhzZXVuZ3RhZS5raW1AcXJhZnRlYy5jb20SEW1sb3BzQHFyYWZ0ZWMuY29tGk1GbHl0ZTogJ2V4YW1wbGVzLmZseXRlX2ZpeGVkcmF0ZS5wb3NpdGl2ZV93ZicgaGFzIHN1Y2NlZWRlZCBpbiAnZGV2ZWxvcG1lbnQnLiLgAkV4ZWN1dGlvbiAnZXhhbXBsZXMuZmx5dGVfZml4ZWRyYXRlLnBvc2l0aXZlX3dmIFtmZWUyMmVjZDM2MDE4NWMzZDAwMF0nIGhhcyBzdWNjZWVkZWQgaW4gJ2RldmVsb3BtZW50Jy4gVmlldyBkZXRhaWxzIGF0CjxhIGhyZWY9XGh0dHA6Ly9mbG93LnFyYWZ0cGlsb3QuY29tL2NvbnNvbGUvcHJvamVjdHMvZmx5dGVzbmFja3MvZG9tYWlucy9kZXZlbG9wbWVudC9leGVjdXRpb25zL2ZlZTIyZWNkMzYwMTg1YzNkMDAwPgpodHRwOi8vZmxvdy5xcmFmdHBpbG90LmNvbS9jb25zb2xlL3Byb2plY3RzL2ZseXRlc25hY2tzL2RvbWFpbnMvZGV2ZWxvcG1lbnQvZXhlY3V0aW9ucy9mZWUyMmVjZDM2MDE4NWMzZDAwMDwvYT4uIAo=" |base64 --decode
seungtae.kim@qraftec.commlops@qraftec.comMFlyte: 'examples.flyte_fixedrate.positive_wf' has succeeded in 'development'."?Execution 'examples.flyte_fixedrate.positive_wf [fee22ecd360185c3d000]' has succeeded in 'development'. View details at
<a href=\<http://flow.qraftpilot.com/console/projects/flytesnacks/domains/development/executions/fee22ecd360185c3d000>>
<http://flow.qraftpilot.com/console/projects/flytesnacks/domains/development/executions/fee22ecd360185c3d000></a>.
katrina
Prafulla Mahindrakar
08/22/2022, 3:58 PMSeungTaeKim
08/23/2022, 6:34 AM{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "sqs:*",
"Resource": "*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "sns:*",
"Resource": "arn:aws:sns:<aws region>:<account>:<sns service name>"
},
{
"Sid": "VisualEditor2",
"Effect": "Allow",
"Action": "ses:*",
"Resource": "*"
}
]
}
we have checked SNS & SQS work without problems before, but don't know why emails not arrived.
I think the policy setting should be added in this docs, https://docs.flyte.org/projects/cookbook/en/latest/auto/deployment/lp_notifications.html
Thank you!Prafulla Mahindrakar
08/23/2022, 6:52 AMSeungTaeKim
08/23/2022, 6:53 AMPrafulla Mahindrakar
08/23/2022, 6:54 AMSeungTaeKim
08/29/2022, 9:49 AMPrafulla Mahindrakar
08/29/2022, 9:56 AM