Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

Hi, is it possible to send Platform Events to something like external Kafka cluster or is AWS SQS/SNS the only available option right now?

Following, this would also be desirable for us

yes just yesterday someone asked :slightly_smiling_face:

Yes, we can send the taskExecution and workflowExecution events to aws SQS. <https://docs.flyte.org/en/latest/deployment/cluster_config/eventing.html>

<@USU6W5ATA> yes, I’m aware - I was just wondering if Kafka is an option rather than aws SQS

Yes, flyte can send a cloudevent to kafka, <https://github.com/flyteorg/flyteadmin/blob/1817f38b977e9e821f732fb439cdd2f0acaead21/pkg/runtime/interfaces/application_configuration.go#L196-L200|config> will be like
```  cloudEvents:
    enable: true
    kafka:
      brokers: flyte
    type: kafka```
sorry, we don’t have doc for it. I’m working on it, will share with you.

Oh nice thanks! How would auth work here?

We have a cluster with kafka running (via Strimzi) and the cluster is gated by AuthService via Emissary Ingress

Which in turn is using AWS Cognito for authentication

Ah interesting <@U03APPBC2SC> - we use Aiven for our hosted Kafka, and we make the credentials available in our cluster via AWS Secrets Manager/`ExternalSecrets` operator

<@U03BT8P2XGA> right thanks - I suppose I’d have to do something similar for our cluster access but how would I use this in the Flyte kafka config?

Not familiar with Cognito, but I think we probably need to update IAM role or add credential for flyteadmin. if flyteadmin has access to connect to the kafka, it should be able to send event to kafka.

I see - it can’t be done right now though I assume?

I'm also very interested in this, also using Aiven

Currently, kafka config only expose topics, brokers without auth stuff. we definitely can add it if needed. Free feel to try this publisher, and let us know what we can help. love to dig into it.