https://flyte.org logo
Title
s

seunggs

11/02/2022, 6:00 PM
Hi, is it possible to send Platform Events to something like external Kafka cluster or is AWS SQS/SNS the only available option right now?
r

Rahul Mehta

11/02/2022, 6:08 PM
Following, this would also be desirable for us
k

Ketan (kumare3)

11/02/2022, 7:01 PM
yes just yesterday someone asked 🙂
k

Kevin Su

11/02/2022, 7:01 PM
Yes, we can send the taskExecution and workflowExecution events to aws SQS. https://docs.flyte.org/en/latest/deployment/cluster_config/eventing.html
s

seunggs

11/02/2022, 7:35 PM
@Kevin Su yes, I’m aware - I was just wondering if Kafka is an option rather than aws SQS
k

Kevin Su

11/02/2022, 7:45 PM
Yes, flyte can send a cloudevent to kafka, config will be like
cloudEvents:
    enable: true
    kafka:
      brokers: flyte
    type: kafka
sorry, we don’t have doc for it. I’m working on it, will share with you.
s

seunggs

11/02/2022, 7:46 PM
Oh nice thanks! How would auth work here?
We have a cluster with kafka running (via Strimzi) and the cluster is gated by AuthService via Emissary Ingress
Which in turn is using AWS Cognito for authentication
r

Rahul Mehta

11/02/2022, 7:47 PM
Ah interesting @seunggs - we use Aiven for our hosted Kafka, and we make the credentials available in our cluster via AWS Secrets Manager/`ExternalSecrets` operator
s

seunggs

11/02/2022, 7:49 PM
@Rahul Mehta right thanks - I suppose I’d have to do something similar for our cluster access but how would I use this in the Flyte kafka config?
k

Kevin Su

11/02/2022, 8:09 PM
Not familiar with Cognito, but I think we probably need to update IAM role or add credential for flyteadmin. if flyteadmin has access to connect to the kafka, it should be able to send event to kafka.
s

seunggs

11/02/2022, 8:42 PM
I see - it can’t be done right now though I assume?
k

Katrina P

11/03/2022, 7:22 PM
I'm also very interested in this, also using Aiven
k

Kevin Su

11/03/2022, 8:21 PM
Currently, kafka config only expose topics, brokers without auth stuff. we definitely can add it if needed. Free feel to try this publisher, and let us know what we can help. love to dig into it.