Hi, is it possible to send Platform Events to some...
# ask-the-community
s
Hi, is it possible to send Platform Events to something like external Kafka cluster or is AWS SQS/SNS the only available option right now?
r
Following, this would also be desirable for us
k
yes just yesterday someone asked 🙂
k
Yes, we can send the taskExecution and workflowExecution events to aws SQS. https://docs.flyte.org/en/latest/deployment/cluster_config/eventing.html
s
@Kevin Su yes, I’m aware - I was just wondering if Kafka is an option rather than aws SQS
k
Yes, flyte can send a cloudevent to kafka, config will be like
Copy code
cloudEvents:
    enable: true
    kafka:
      brokers: flyte
    type: kafka
sorry, we don’t have doc for it. I’m working on it, will share with you.
s
Oh nice thanks! How would auth work here?
We have a cluster with kafka running (via Strimzi) and the cluster is gated by AuthService via Emissary Ingress
Which in turn is using AWS Cognito for authentication
r
Ah interesting @seunggs - we use Aiven for our hosted Kafka, and we make the credentials available in our cluster via AWS Secrets Manager/`ExternalSecrets` operator
s
@Rahul Mehta right thanks - I suppose I’d have to do something similar for our cluster access but how would I use this in the Flyte kafka config?
k
Not familiar with Cognito, but I think we probably need to update IAM role or add credential for flyteadmin. if flyteadmin has access to connect to the kafka, it should be able to send event to kafka.
s
I see - it can’t be done right now though I assume?
k
I'm also very interested in this, also using Aiven
k
Currently, kafka config only expose topics, brokers without auth stuff. we definitely can add it if needed. Free feel to try this publisher, and let us know what we can help. love to dig into it.
201 Views