🤕 Issue: Slack notifications in EKS. Hi everyone🙌 , we have deployed Flyte in EKS and we are trying to implement Slack notifications. We have created a SNS topic with a SQS queue subscribed, and the flyte admin role has permissions to read/write to them. The flyte-core pods run with no problem, but when we run a launch plan configured as follow:

from flytekit import LaunchPlan, Slack, WorkflowExecutionPhase
from flytekit.configuration import Config
from flytekit.remote import (
    FlyteRemote,
)

from workflow import simple_wf

remote = FlyteRemote(config=Config.auto(), default_project="flyteexamples", default_domain="development")

project_launch_plan = LaunchPlan.create(
    name="notifications-test-2",
    workflow=simple_wf,
    notifications=[
        Slack(
            phases=[WorkflowExecutionPhase.SUCCEEDED],
            recipients_email=["<slack channel email>"],
        ),
        Slack(
            phases=[
                WorkflowExecutionPhase.FAILED,
                WorkflowExecutionPhase.ABORTED,
                WorkflowExecutionPhase.TIMED_OUT,
            ],
            recipients_email=["<slack channel email>"],
        ),
    ]
)

registered_launch_plan = remote.register_launch_plan(
    entity=project_launch_plan, version="ojoNW9eBpiTeb_4h0ptpHw==", project="flyteexamples", domain="development"
)

remote.client.update_launch_plan(registered_launch_plan.id, "ACTIVE")

We do not see any msg to our channel and SQS topic 🥲. Our chart configuration is as follows:

workflow_notifications:
  enabled: true
  config:
    notifications:
      type: aws
      region: "{{ .Values.userSettings.accountRegion }}"
      publisher:
        topicName: "{{ .Values.userSettings.snsTopicNameArn }}"
      processor:
        queueName: "{{ .Values.userSettings.sqsQueueName }}"
        accountId: "{{ .Values.userSettings.accountNumber }}"
      emailer:
        subject: "Flyte: {{ project }}/{{ domain }}/{{ launch_plan.name }} has {{ phase }}"
        sender: ""
        body: >
          Execution \"{{ workflow.name }} [{{ name }}]\" has {{ phase }} in \"{{ domain }}\". View details at
          <a href=\<http://flyte.company.com/console/projects/{{> project }}/domains/{{ domain }}/executions/{{ name }}>
          <http://flyte.company.com/console/projects/{{> project }}/domains/{{ domain }}/executions/{{ name }}</a>. {{ error }}

The logs of flyteadmin.

Defaulted container "flyteadmin" out of: flyteadmin, run-migrations (init), seed-projects (init), sync-cluster-resources (init), generate-secrets (init)
time="2023-10-03T15:10:15Z" level=info msg="Using config file: [/etc/flyte/config/cluster_resources.yaml /etc/flyte/config/clusters.yaml /etc/flyte/config/db.yaml /etc/flyte/config/domain.yaml /etc/flyte/config/notifications.yaml /etc/flyte/config/remoteData.yaml /etc/flyte/config/server.yaml /etc/flyte/config/storage.yaml /etc/flyte/config/task_resource_defaults.yaml]"
{"json":{},"level":"warning","msg":"stow configuration section missing, defaulting to legacy s3/minio connection config","ts":"2023-10-03T15:10:16Z"}
{"json":{},"level":"warning","msg":"Starting notifications processor","ts":"2023-10-03T15:10:16Z"}
{"json":{"exec_id":"afcxqsptn9ljrc2f4xcg"},"level":"warning","msg":"Failed to fetch override values when assigning task resource default values for [resource_type:WORKFLOW project:\"flyteexamples\" domain:\"development\" name:\"basics.workflow.simple_wf\" version:\"ojoNW9eBpiTeb_4h0ptpHw==\" ]: Resource [{Project:flyteexamples Domain:development Workflow:basics.workflow.simple_wf LaunchPlan: ResourceType:TASK_RESOURCE}] not found","ts":"2023-10-03T15:11:16Z"}
{"json":{"exec_id":"afcxqsptn9ljrc2f4xcg"},"level":"warning","msg":"Failed to fetch override values when assigning execution queue for [{ResourceType:WORKFLOW Project:flyteexamples Domain:development Name:basics.workflow.simple_wf Version:ojoNW9eBpiTeb_4h0ptpHw== XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] with err: Resource [{Project:flyteexamples Domain:development Workflow:basics.workflow.simple_wf LaunchPlan: ResourceType:EXECUTION_QUEUE}] not found","ts":"2023-10-03T15:11:16Z"}
{"json":{"exec_id":"afcxqsptn9ljrc2f4xcg"},"level":"warning","msg":"Setting security context from auth Role","ts":"2023-10-03T15:11:17Z"}
{"json":{},"level":"warning","msg":"node execution is missing StartedAt","ts":"2023-10-03T15:11:22Z"}

Where do we can see any log to debug this? thank youuu ❤️

If you haven't found the error, you can follow the contribution guide and print the logs under the directory. flyteadmin notifications: https://github.com/flyteorg/flyteadmin/tree/master/pkg/async/notifications contribution guide: https://docs.flyte.org/en/latest/community/contribute.html#how-to-setup-dev-environment-for-flyteidl-f[…]-flytepropeller-datacatalog-and-flytestdlib