Hi 🙌we are trying to implement workflow Slack notifications. We have deployed Flyte in EKS with the flyte-binary chart. Our workflow has the notification set:

project_launch_plan = LaunchPlan.create(
    name=f"{schedule_config.name}-{pipeline.name}",
    workflow=pipeline,
    default_inputs=schedule_config.default_inputs,
    fixed_inputs=schedule_config.fixed_inputs,
    schedule=schedule_config.schedule,
    notifications=[
        Slack(
            phases=[
                WorkflowExecutionPhase.FAILED,
                WorkflowExecutionPhase.SUCCEEDED,
                WorkflowExecutionPhase.ABORTED,
                WorkflowExecutionPhase.TIMED_OUT,
            ],
            recipients_email=["<slack channel email>"],
        )
    ],
)

But in our slack channel nothing appears when the launch plan succeeds. In the documentation there is this page that explains how to configure the FlyteAdmin, how we can do that with the flyte-binary chart? Also we have seen this pr to add webhook option, will this allow us to send notifications without creating extra infrastructure?

i would not rely on the pr for now. the existing notifications mechanism should work fine.

Thank youu! I will try this 🙌

I'm also curious how I can debug notifications, I don't see any logs in the flyte pod regarding workflow notification delivery failures, and the workflow completes yet I never receive any emails indicating its completion status. I have things set up very similarly to how @Marti Jorda Roca does here by the looks of it, and am seeing the same workflow is not defined error as them.

For what I have seen, notifications work only wiht the chart flyte-core an not the binary

Does anyone know the difference between

workflow_notifications

and

notifications

sections? Thanks! (I only see workflow_notifications here as an option in flyte-core)

Yeah, I see the same

parse error at (flyte-binary/templates/deployment.yaml:4): function "workflow" not defined

error with the

flyte-binary

chart, even putting it

inline

as @Yee suggested above yields this error (I tried both

workflow-notifications

and

notifications

). Is there any way to get notifications going with just the

flyte-binary

? Does anyone have a functioning example?

it should work in either case - from any chart, you need to make sure that ultimately the admin component can pick it up.

I'm able to successfully send test SNS/SQS messages via AWS, something tricky going on with

flyte-binary

and notifications though. Do you see something wrong with this indentation at first glance?

configuration:
  database:
    password: ***************
    host: **************
    dbname: some-name
  storage:
    metadataContainer: some-name-dev-meta
    userDataContainer: some-name-dev-user
    provider: s3
    providerConfig:
      s3:
        region: "us-east-1"
        authType: "iam"
  inline:
    plugins:
      k8s:
        inject-finalizer: true
        default-env-vars:
          - AWS_METADATA_SERVICE_TIMEOUT: #
          - AWS_METADATA_SERVICE_NUM_ATTEMPTS: ##
    storage:
      cache:
        max_size_mbs: ###
        target_gc_percent: ###
    tasks:
      task-plugins:
        enabled-plugins:
          - container
          - sidecar
          - k8s-array
        default-for-task-types:
          - container: container
          - sidecar: sidecar
          - container_array: k8s-array
    notifications:
      enabled: true
      config:
        notifications:
          type: "aws"
          region: "us-east-1"
          publisher:
            topicName: "arn:aws:sns:us-east-1:############:some-name"
          processor:
            queueName: "some-name"
            accountId: "############"
          emailer:
            subject: 'Notice: Execution "{{ workflow.name }}" has {{ phase }} in "{{ domain }}".'
            sender: "<mailto:email@address.com|email@address.com>"
            body: >
              Execution \"{{ workflow.name }} [{{ name }}]\" has {{ phase }} in \"{{ domain }}\". View details at
              <a href=<http://localhost:8088/console/projects/{{> project }}/domains/{{ domain }}/executions/{{ name }}>
              <http://localhost:8088/console/projects/{{> project }}/domains/{{ domain }}/executions/{{ name }}</a>. {{ error }}

serviceAccount:
  create: true
  annotations:
    <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: "arn:aws:iam::############:role/#####-#########"

can you try deploying without the template for now?

I can see this for my inline config:

# cat 010-inline-config.yaml
notifications:
  config:
    notifications:
      emailer:
        body: hi this is the email body
        sender: <mailto:email@address.com|email@address.com>
        subject: hi this is the email subject
      processor:
        accountId: "###"
        queueName: some-name
      publisher:
        topicName: arn:aws:sns:us-east-1:##:some-name
      region: us-east-1
      type: aws
  enabled: true
plugins:
  k8s:
    default-env-vars:
    - AWS_METADATA_SERVICE_TIMEOUT: #
    - AWS_METADATA_SERVICE_NUM_ATTEMPTS: ##
    inject-finalizer: true
storage:
  cache:
    max_size_mbs: ###
    target_gc_percent: ###
tasks:
  task-plugins:
    default-for-task-types:
    - container: container
    - sidecar: sidecar
    - container_array: k8s-array
    enabled-plugins:
    - container
    - sidecar
    - k8s-array

and still not working?

This format does indeed at least attempt to access your queue and topic for `flyte-binary`:

configuration:
  inline:
    notifications:
      type: "aws"
      region: "us-east-1"
      publisher:
        topicName: "arn:aws:sns:us-east-1:#########:some-name"
      processor:
        queueName: "some-name"
        accountId: "#########"
      emailer:
        sender: "<mailto:email@address.com|email@address.com>"
        subject: "hi this is the email subject"
        body: "hi this is the email body"

Unfortunately if we switch to the more complex syntax (with

{{ workflow }}

), we still see

function "workflow" not defined

errors. Still trying to debug that side of things. Thank you again for all of your help @Yee, truly appreciate your time and effort in this thread.

just stick to the normal email body for now.

Yes, I'm using SES for email.

any luck with this today?

Still no luck unfortunately, thanks for continuing to dig into this! Yeah, I was looking at that code block, from what I can tell I don't see how it could get sent out without being decoded from base64 first. 🤔

added some debug logging here: https://github.com/flyteorg/flyteadmin/pull/614, puling it into here: https://github.com/flyteorg/flyte/pull/4053 and building a single binary image from that branch here: https://github.com/flyteorg/flyte/actions/runs/6242719645

correct

Thanks for spinning up this debug build for me @Yee I'm pulling it in via this (correct me if this is wrong?):

image:
  repository: <http://ghcr.io/flyteorg/flyte-sandbox-bundled|ghcr.io/flyteorg/flyte-sandbox-bundled>
  tag: sha-36d4f2fa621ac94028c060f4a45276817cee617f

In the logs I see this (weirdly nothing about base64, which makes me suspect that I might not have the write repository?):

{"json":{"src":"composite_workqueue.go:88"},"level":"debug","msg":"Subqueue handler batch round","ts":"2023-09-20T17:15:24Z"}
{"json":{"src":"composite_workqueue.go:98"},"level":"debug","msg":"Dynamically configured batch size [-1]","ts":"2023-09-20T17:15:24Z"}
{"json":{"src":"composite_workqueue.go:129"},"level":"debug","msg":"Exiting SubQueue handler batch round","ts":"2023-09-20T17:15:24Z"}
{"json":{"src":"aws_emailer.go:63"},"level":"debug","msg":"Sent email to [email@address.com] sub: hi this is the subject","ts":"2023-09-20T17:15:24Z"}
{"json":{"src":"composite_workqueue.go:88"},"level":"debug","msg":"Subqueue handler batch round","ts":"2023-09-20T17:15:25Z"}
{"json":{"src":"composite_workqueue.go:98"},"level":"debug","msg":"Dynamically configured batch size [-1]","ts":"2023-09-20T17:15:25Z"}
{"json":{"src":"composite_workqueue.go:129"},"level":"debug","msg":"Exiting SubQueue handler batch round","ts":"2023-09-20T17:15:25Z"}
{"json":{"src":"composite_workqueue.go:88"},"level":"debug","msg":"Subqueue handler batch round","ts":"2023-09-20T17:15:26Z"}
{"json":{"src":"composite_workqueue.go:98"},"level":"debug","msg":"Dynamically configured batch size [-1]","ts":"2023-09-20T17:15:26Z"}
{"json":{"src":"composite_workqueue.go:129"},"level":"debug","msg":"Exiting SubQueue handler batch round","ts":"2023-09-20T17:15:26Z"}

The fact that this isn't base64 encoded makes me think it could be a problem on the AWS side? But that doesn't explain why sending test AWS messages do not come through as base64 encoded, they come through as the inputted message as expected...

that’s the correct image yeah - can you confirm that that’s what the pod is actually running?

Is this image built the exact same way as the default images? With the default repo with

latest

tag we can deploy Flyte just fine but with that image we fail readiness checks:

deployment:
  image:
    pullPolicy: IfNotPresent
    repository: "<http://cr.flyte.org/flyteorg/flyte-binary|cr.flyte.org/flyteorg/flyte-binary>" # default image works fine
    tag: "latest"
    # repository: <http://ghcr.io/flyteorg/flyte-sandbox-bundled|ghcr.io/flyteorg/flyte-sandbox-bundled>
    # tag: sha-36d4f2fa621ac94028c060f4a45276817cee617f

Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  3m39s                  default-scheduler  Successfully assigned am-flyte-dev/am-flyte-flyte-binary-8596c6cf56-v2ncj to ip-###.###.###.###.ec2.internal
  Normal   Pulled     3m38s                  kubelet            Container image "postgres:15-alpine" already present on machine
  Normal   Created    3m38s                  kubelet            Created container wait-for-db
  Normal   Started    3m38s                  kubelet            Started container wait-for-db
  Warning  BackOff    3m14s                  kubelet            Back-off restarting failed container
  Normal   Pulled     3m8s (x3 over 3m37s)   kubelet            Container image "<http://ghcr.io/flyteorg/flyte-sandbox-bundled:sha-36d4f2fa621ac94028c060f4a45276817cee617f|ghcr.io/flyteorg/flyte-sandbox-bundled:sha-36d4f2fa621ac94028c060f4a45276817cee617f>" already present on machine
  Normal   Created    3m8s (x3 over 3m36s)   kubelet            Created container flyte
  Normal   Started    3m8s (x3 over 3m36s)   kubelet            Started container flyte
  Warning  Unhealthy  2m58s (x9 over 3m35s)  kubelet            Readiness probe failed: Get "<http://192.168.108.142:8088/healthcheck>": dial tcp ###.###.###.###:8088: connect: connection refused
  Warning  Unhealthy  2m58s (x3 over 3m28s)  kubelet            Liveness probe failed: Get "<http://192.168.108.142:8088/healthcheck>": dial tcp ###.###.###.###:8088: connect: connection refused

built exactly the same way.

Here are the additional logs, does anything stand out to you @Yee?:

{
    "json": {
        "src": "aws_processor.go:46"
    },
    "level": "info",
    "msg": "debugb64 Original stringMsg [{\n  \"Type\" : \"Notification\",\n  \"MessageId\" : \"a85d47a2-1325-547d-ae72-1a2485c93d77\",\n  \"TopicArn\" : \"arn:aws:sns:us-east-1:##########:am-flyte\",\n  \"Subject\" : \"flyteidl.admin.EmailNotification\",\n  \"Message\" : \"CiJwZXRlci5rbGluZ2VsaG9mZXJAZW5lcmd5dmF1bHQuY29tEiJwZXRlci5rbGluZ2VsaG9mZXJAZW5lcmd5dmF1bHQuY29tGhZoaSB0aGlzIGlzIHRoZSBzdWJqZWN0IhNoaSB0aGlzIGlzIHRoZSBib2R5\",\n  \"Timestamp\" : \"2023-09-21T19:08:28.909Z\",\n  \"SignatureVersion\" : \"1\",\n  \"Signature\" : \"bIDTSNY2vjvRX7aX17oNTKH9Xh3ckqmy30XB/YU68TAlr42cIsuTDb0xS68V/n3KhrWlhxo4JPRHWLEdy4kP+jWO152M4tud5rx26jpKh0GH5tDhzt0rzebKrLdodNOnIyM5SKb8MzUZ36I699JFnIw8wtVkfUfgN6ciRIxTJAq9HqDq8f3DwLOu/OKl3SZSKkvRJecDxXgRdhHAA3oLCcpBVZvdG2UVikPNpHxJfzzl4V6IUJ7Ze30UWTr3BTHVWbBUsbcrR8J5jTMDjmSr+pUD8iU/ppJ3AXHc4PcM/3uUQCDYxMg3Pl6PXFLpg1C28v0ikKBBCbpUeC1/BrDtbQ==\",\n  \"SigningCertURL\" : \"<https://sns.us-east-1.amazonaws.com/SimpleNotificationService-#########.pem>\",\n  \"UnsubscribeURL\" : \"<https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe>\u0026SubscriptionArn=arn:aws:sns:us-east-1:##########:am-flyte:b######-860f-414d-bac4-5fe3e7dd2186\"\n}]",
    "ts": "2023-09-21T19:08:28Z"
}
{
    "json": {
        "src": "aws_processor.go:59"
    },
    "level": "info",
    "msg": "debugb64 snsJSONFormat [map[Message:CiJwZXRlci5rbGluZ2VsaG9mZXJAZW5lcmd5dmF1bHQuY29tEiJwZXRlci5rbGluZ2VsaG9mZXJAZW5lcmd5dmF1bHQuY29tGhZoaSB0aGlzIGlzIHRoZSBzdWJqZWN0IhNoaSB0aGlzIGlzIHRoZSBib2R5 MessageId:a85d47a2-1325-547d-ae72-1a2485c93d77 Signature:bIDTSNY2vjvRX7aX17oNTKH9Xh3ckqmy30XB/YU68TAlr42cIsuTDb0xS68V/n3KhrWlhxo4JPRHWLEdy4kP+jWO152M4tud5rx26jpKh0GH5tDhzt0rzebKrLdodNOnIyM5SKb8MzUZ36I699JFnIw8wtVkfUfgN6ciRIxTJAq9HqDq8f3DwLOu/OKl3SZSKkvRJecDxXgRdhHAA3oLCcpBVZvdG2UVikPNpHxJfzzl4V6IUJ7Ze30UWTr3BTHVWbBUsbcrR8J5jTMDjmSr+pUD8iU/ppJ3AXHc4PcM/3uUQCDYxMg3Pl6PXFLpg1C28v0ikKBBCbpUeC1/BrDtbQ== SignatureVersion:1 SigningCertURL:<https://sns.us-east-1.amazonaws.com/SimpleNotificationService-##########.pem> Subject:flyteidl.admin.EmailNotification Timestamp:2023-09-21T19:08:28.909Z TopicArn:arn:aws:sns:us-east-1:##########:am-flyte Type:Notification UnsubscribeURL:<https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe>\u0026SubscriptionArn=arn:aws:sns:us-east-1:##########:am-flyte:######-860f-414d-bac4-5fe3e7dd2186]]",
    "ts": "2023-09-21T19:08:28Z"
}
{
    "json": {
        "src": "aws_processor.go:88"
    },
    "level": "info",
    "msg": "debugb64 Decoded valueString [CiJwZXRlci5rbGluZ2VsaG9mZXJAZW5lcmd5dmF1bHQuY29tEiJwZXRlci5rbGluZ2VsaG9mZXJAZW5lcmd5dmF1bHQuY29tGhZoaSB0aGlzIGlzIHRoZSBzdWJqZWN0IhNoaSB0aGlzIGlzIHRoZSBib2R5] to [[10 34 112 101 116 101 114 46 107 108 105 110 103 101 108 104 111 102 101 114 64 101 110 101 114 103 121 118 97 117 108 116 46 99 111 109 18 34 112 101 116 101 114 46 107 108 105 110 103 101 108 104 111 102 101 114 64 101 110 101 114 103 121 118 97 117 108 116 46 99 111 109 26 22 104 105 32 116 104 105 115 32 105 115 32 116 104 101 32 115 117 98 106 101 99 116 34 19 104 105 32 116 104 105 115 32 105 115 32 116 104 101 32 98 111 100 121]]",
    "ts": "2023-09-21T19:08:28Z"
}
{
    "json": {
        "src": "aws_emailer.go:54"
    },
    "level": "info",
    "msg": "debugb64 Sending email hi this is the body",
    "ts": "2023-09-21T19:08:28Z"
}
{
    "json": {
        "src": "aws_emailer.go:64"
    },
    "level": "debug",
    "msg": "Sent email to [email@address.com] sub: hi this is the subject",
    "ts": "2023-09-21T19:08:29Z"
}

Hi, I am using the flyte-core chart and the flyteadmin pods raises this error:

InvalidAddress: The address <https://sqs.eu-west-1.amazonaws.com/> is not valid for this endpoint.\n\tstatus code: 404

Any idea what is wrong? PD: sorry for this subject that it is not related to what your are currently talking.

@Marti Jorda Roca not sure if related, some recent changes to the notifications config for

flyte-core

in this PR, but I don't think there's been a new release with this included yet

thanks I will try to downgrade the helm version

@Marti Jorda Roca let’s start a different thread for that.

@Yee thanks for looking into it. Not too worried about base64 decoding it if I have to, but the fact that I can’t use ‘workflow’ and those other variables are undefined in the helm chart is the main thing blocking notifications being useful over here. Any ideas as to how we could debug that, since it seems to be appearing inline as we would expect?

@Peter Klingelhofer sorry what are you talking about? is there a missing helm chart configuration?

@Yee it’s the same error that @Marti Jorda Roca was originally getting (hence why I chimed in in this thread 😀) - ‘function workflow is not defined’ if you try to use the ‘workflow’ (or any of the other variables in the docs) in the template literal (like the example in the docs).

new thread plz

Solution via escaping brackets here in the new thread: https://flyte-org.slack.com/archives/CP2HDHKE1/p1695826300502159