Hi, is it possible to send Platform Events to something like external Kafka cluster or is AWS SQS/SNS the only available option right now?

Following, this would also be desirable for us

yes just yesterday someone asked 🙂

Yes, we can send the taskExecution and workflowExecution events to aws SQS. https://docs.flyte.org/en/latest/deployment/cluster_config/eventing.html

@Kevin Su yes, I’m aware - I was just wondering if Kafka is an option rather than aws SQS

Yes, flyte can send a cloudevent to kafka, config will be like

cloudEvents:
    enable: true
    kafka:
      brokers: flyte
    type: kafka

sorry, we don’t have doc for it. I’m working on it, will share with you.

Oh nice thanks! How would auth work here?

Ah interesting @seunggs - we use Aiven for our hosted Kafka, and we make the credentials available in our cluster via AWS Secrets Manager/`ExternalSecrets` operator

@Rahul Mehta right thanks - I suppose I’d have to do something similar for our cluster access but how would I use this in the Flyte kafka config?

Not familiar with Cognito, but I think we probably need to update IAM role or add credential for flyteadmin. if flyteadmin has access to connect to the kafka, it should be able to send event to kafka.

I see - it can’t be done right now though I assume?

I'm also very interested in this, also using Aiven

Currently, kafka config only expose topics, brokers without auth stuff. we definitely can add it if needed. Free feel to try this publisher, and let us know what we can help. love to dig into it.