#4045 Sandbox notifications - not sending to slack Issue created by dyu-bot Describe the bug I am testing out the notifications sandbox configuration with this config: I set up a sendgrid account with a verified sender email address, and use that api key in the configs.

workflow_notifications:
    enabled: true
    config:
      notifications:
        type: sandbox
        region: foo (<- I had to set this, otherwise helm was complaining) 
        emailer:
          emailServerConfig:
            serviceName: sendgrid
            apiKeyEnvVar: $SENDGRID_API_KEY
          subject: "Notice: Execution \"{{ workflow.name }}\" has {{ phase }} in \"{{ domain }}\"."
          sender:  "xxx@xxx.com"
          body: >
             Execution \"{{ workflow.name }} [{{ name }}]\" has {{ phase }} in \"{{ domain }}\". View details at
             <a href=\https://xxx.com/console/projects/{{ project }}/domains/{{ domain }}/executions/{{ name }}>
             <https://xxx.com/console/projects/>{{ project }}/domains/{{ domain }}/executions/{{ name }}</a>. {{ error }}

I'm using

flyte-core

helm chart

v1.6.1

but overriding the flyteadmin image to

v1.1.119

to be able to use the sandbox processor and publisher

flyteadmin:
    image:
      repository: "<http://ghcr.io/flyteorg/flyteadmin|ghcr.io/flyteorg/flyteadmin>"
      tag: "v1.1.119"

Flyte is deployed on GKE 1.26 I'm running this sample workflow in the docs to notify upon success:

@task
def double_int_and_print(a: int) -> str:
    return str(a * 2)


@workflow
def int_doubler_wf(a: int) -> str:
    doubled = double_int_and_print(a=a)
    return doubled

wacky_int_doubler_lp = LaunchPlan.get_or_create(
    name="wacky_int_doubler",
    workflow=int_doubler_wf,
    default_inputs={"a": 4},
    notifications=[
        Slack(
            phases=[
                WorkflowExecutionPhase.SUCCEEDED,
                WorkflowExecutionPhase.ABORTED,
                WorkflowExecutionPhase.TIMED_OUT,
                WorkflowExecutionPhase.FAILED,
            ],
            recipients_email=["<http://xxx.slack.com|xxx.slack.com>"],
        ),
    ],
)

In the logs, I do see the publisher publishing the message, but there is an error running the processor:

{"json":{"src":"sandbox_processor.go:21"},"level":"warning","msg":"Starting SandBox notifications processor","ts":"2023-09-18T09:02:23Z"}
{"json":{"src":"sandbox_processor.go:46"},"level":"debug","msg":"no message to process","ts":"2023-09-18T09:02:23Z"}
{"json":{"src":"sandbox_processor.go:23"},"level":"error","msg":"error with running processor err: [\u003cnil\u003e] ","ts":"2023-09-18T09:02:23Z"}
{"json":{"exec_id":"xxx","src":"sandbox_publisher.go:15"},"level":"debug","msg":"Publishing the following message [recipients_email:\"<http://xxx.com|xxx.com>\" sender_email:\"xxx\" subject_line:\"Notice: Execution \\\"xxx\\\" has succeeded in \\\"development\\\".\" body:\"Execution \\\\\\\"xxx.smoke.int_doubler_wf [xxx]\\\\\\\" has succeeded in \\\\\\\"development\\\\\\\". View details at \u003ca href=\\\\<https://xxx.com/console/projects/xxx/domains/development/executions/xxx\u003e> <https://xxx.com/console/projects/xxx/domains/development/executions/xxx\u003c/a\u003e>. \\n\" ]","ts":"2023-09-18T09:02:26Z"}

Also, upon checking the sendgrid account, it did not attempt to send any email, so it looks like it isn't reaching sendgrid (probably due to the processor not working). Expected behavior The message to reach slack. Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte