https://flyte.org
Join Slack
Hi! Does `flyte-core` chart support sendgrid based `notifications` ? Docs <here> say specify it can...
# flyte-support
s
Hi! Does 
flyte-core
chart support sendgrid based 
notifications
? Docs here say specify it can be configured for sandbox, but when I checked flyte-core template it seems to only support the AWS setup, and under a 
workflow_notifications
section rather than 
notifications
.
h
Hey @straight-businessperson-54649, yes it does. It's indeed under 
workflow_notifications
in flyte-core (might be inconsistent) but you can put the same 
emailer
block there and it will work just the same. Have you successfully tested it in sandbox?
s
No, not tested yet. Was just curious. Will test it out, thanks @high-park-82026
Hey @high-park-82026 I tested. I tried using the 
sendgrid
setup with 
flyte-core
chart as such:
Copy code
workflow_notifications:
    enabled: true
    config:
      notifications:
        type: sandbox
        emailer:
          emailServerConfig:
            serviceName: sendgrid
            apiKeyEnvVar: $SENDGRID_API_KEY
          subject: "Notice: Execution \"{{ workflow.name }}\" has {{ phase }} in \"{{ domain }}\"."
          sender:  "<mailto:foo@foo.com|foo@foo.com>"
          body: >
             asdf
When upgrading the chart, this error appeared:
Copy code
helm.go:81: [debug] template: flyte/charts/flyte-core/templates/flytescheduler/deployment.yaml:16:27: executing "flyte/charts/flyte-core/templates/flytescheduler/deployment.yaml" at <include (print .Template.BasePath "/admin/configmap.yaml") .>: error calling include: template: flyte/charts/flyte-core/templates/admin/configmap.yaml:48:28: executing "flyte/charts/flyte-core/templates/admin/configmap.yaml" at <.Values.workflow_notifications.config.notifications.region>: wrong type for value; expected string; got interface {}
UPGRADE FAILED
Which suggests that the helm chart expects the fields for the 
AWS setup
(region, publisher, processer) and not the 
sendgrid
setup, which doesn't have those fields.
cc: @thankful-minister-83577 as well, if you know anything, as saw you're also on this topic!
Hey bumping this, thanks!
t
sorry in the middle of a ton of things… will respond later today
🙏 1
gratitude thank you 1
h
Hey @straight-businessperson-54649, have you attempted to set region?
Copy code
workflow_notifications:
    enabled: true
    config:
      notifications:
        type: sandbox
        region: wrong-region
        emailer:
          emailServerConfig:
            serviceName: sendgrid
            apiKeyEnvVar: $SENDGRID_API_KEY
          subject: "Notice: Execution \"{{ workflow.name }}\" has {{ phase }} in \"{{ domain }}\"."
          sender:  "<mailto:foo@foo.com|foo@foo.com>"
          body: >
             asdf
This error is coming from helm chart, I can dig up why it's throwing that but just want to see if we can get past this
t
s
Hey all thanks for the reply! I tried adding a dummy region and its able to deploy now 🙂 However 
sendgrid
never attempts to send an email, and I don't see any evidence of flyteadmin trying to reach sendgrid in the logs (grepped on keywords 
sendgrid
and 
notification
) when running a successful execution with the following launchplan copied directly from the example in the docs:
Copy code
@task
def double_int_and_print(a: int) -> str:
    return str(a * 2)


@workflow
def int_doubler_wf(a: int) -> str:
    doubled = double_int_and_print(a=a)
    return doubled

wacky_int_doubler_lp = LaunchPlan.get_or_create(
    name="wacky_int_doubler",
    workflow=int_doubler_wf,
    default_inputs={"a": 4},
    notifications=[
        Slack(
            phases=[
                WorkflowExecutionPhase.SUCCEEDED,
                WorkflowExecutionPhase.ABORTED,
                WorkflowExecutionPhase.TIMED_OUT,
                WorkflowExecutionPhase.FAILED,
            ],
            recipients_email=["<http://xxx.slack.com|xxx.slack.com>"],
        ),
    ],
)
In 
flyteadmin
I see the correct content passed to 
/etc/flyte/config/notifications.yaml
Copy code
notifications:
  type: sandbox
  region: foo
  emailer:
    body: |
      Execution \"{{ workflow.name }} [{{ name }}]\" has {{ phase }} in \"{{ domain }}\". View details at <a href=\<https://xxx.xxxx.com/console/projects/{{> project }}/domains/{{ domain }}/executions/{{ name }}> <https://xxx.xxx.com/console/projects/{{> project }}/domains/{{ domain }}/executions/{{ name }}</a>. {{ error }}
    emailServerConfig:
      apiKeyEnvVar: xxxx
      serviceName: sendgrid
    sender: <mailto:myemail@xxx.com|myemail@xxx.com> (<- this email is my verified sender identity on sendgrid) 
    subject: 'Notice: Execution "{{ workflow.name }}" has {{ phase }} in "{{ domain }}".'
Is there anything else you recommend I check? Do these configurations looks correct? Thanks!
Is there someone I can pair with to debug? Still not sure why it's not hitting sendgrid
t
Copy code
logger:
  show-source: true
  level: 5
should be at the top level of a config yaml file read by admin
s
thanks for the help @thankful-minister-83577!
hey @thankful-minister-83577 I wanted to follow up on what we discussed about the sandbox (sendgrid) implementation needing gcp pub sub. I checked and it seems like actually the processor and publisher for sandbox (see this PR) actually doesn't rely on any cloud pub sub? These were the only fields used in the notifications config (no specifications on publisher, processor, etc):
Copy code
notifications:
  type: sandbox
  emailer:
    emailServerConfig:
      apiKeyEnvVar: "SENDGRID_API_KEY"
    subject: "Notice: Execution \"{{ name }}\" has {{ phase }} in \"{{ domain }}\"."
    sender: "<mailto:example@gmail.com|example@gmail.com>"  # Consider using a generic email instead of personal
    body: >
      Execution \"{{ name }}\" has {{ phase }} in \"{{ domain }}\". View details at
      <a href=\<http://example.com/projects/{{> project }}/domains/{{ domain }}/executions/{{ name }}>
      <http://example.com/projects/{{> project }}/domains/{{ domain }}/executions/{{ name }}</a>. {{ error }}
I summarized the thread in an issue here
t
i’m confused
i thought you were doing this on gcp.
not sandbox
s
Yeah no, I've been trying 
type: sandbox
t
i think you need gcp
s
I guess that was my question initially, was whether sandbox was a working implementation. So guessing not then?
t
don’t rely on sandbox
use pubsub
you need to hit the gizmo library
s
ok got it, I'll try gcp next. Thanks
t
before messing around with emailing, first confirm that you’re seeing messages come through just by looking at the gcp console
👍 1
s
Also if sandbox is out of date, might be good to have the docs updated
After merging the helm fix, the next issue to arise is that the Published message seems to be missing the body, and the Processor (subscriber) cannot pull it. Opened an issue here
25 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by