Hi, is it possible to send Platform Events to something like external Kafka cluster or is AWS SQS/SN...

sticky-angle-28419

11/02/2022, 6:00 PM

Hi, is it possible to send Platform Events to something like external Kafka cluster or is AWS SQS/SNS the only available option right now?

elegant-australia-91422

11/02/2022, 6:08 PM

Following, this would also be desirable for us

freezing-airport-6809

11/02/2022, 7:01 PM

yes just yesterday someone asked 🙂

glamorous-carpet-83516

11/02/2022, 7:01 PM

Yes, we can send the taskExecution and workflowExecution events to aws SQS. https://docs.flyte.org/en/latest/deployment/cluster_config/eventing.html

sticky-angle-28419

11/02/2022, 7:35 PM

@glamorous-carpet-83516 yes, I’m aware - I was just wondering if Kafka is an option rather than aws SQS

👍 1

glamorous-carpet-83516

11/02/2022, 7:45 PM

Yes, flyte can send a cloudevent to kafka, config will be like

Copy code

cloudEvents:
    enable: true
    kafka:
      brokers: flyte
    type: kafka

sorry, we don’t have doc for it. I’m working on it, will share with you.

🙏 1

sticky-angle-28419

11/02/2022, 7:46 PM

Oh nice thanks! How would auth work here?

sticky-angle-28419

11/02/2022, 7:46 PM

We have a cluster with kafka running (via Strimzi) and the cluster is gated by AuthService via Emissary Ingress

sticky-angle-28419

11/02/2022, 7:46 PM

Which in turn is using AWS Cognito for authentication

elegant-australia-91422

11/02/2022, 7:47 PM

Ah interesting @sticky-angle-28419 - we use Aiven for our hosted Kafka, and we make the credentials available in our cluster via AWS Secrets Manager/`ExternalSecrets` operator

sticky-angle-28419

11/02/2022, 7:49 PM

@elegant-australia-91422 right thanks - I suppose I’d have to do something similar for our cluster access but how would I use this in the Flyte kafka config?

glamorous-carpet-83516

11/02/2022, 8:09 PM

Not familiar with Cognito, but I think we probably need to update IAM role or add credential for flyteadmin. if flyteadmin has access to connect to the kafka, it should be able to send event to kafka.

sticky-angle-28419

11/02/2022, 8:42 PM

I see - it can’t be done right now though I assume?

limited-dog-47035

11/03/2022, 7:22 PM

I'm also very interested in this, also using Aiven

glamorous-carpet-83516

11/03/2022, 8:21 PM

Currently, kafka config only expose topics, brokers without auth stuff. we definitely can add it if needed. Free feel to try this publisher, and let us know what we can help. love to dig into it.

241 Views

Open in Slack

Previous Next

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.