Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

Hello there,
I would like to get events equivalent to what is generated by  `flytectl get execution -p flytesnacks -d development -o json`  in the logs or somewhere for post-processing is there a way to do that? (I already have External events configured and processed by a lambda but I missing some context from it)

Yep, but I wanted to stream or react on those events but you’re right I can enrich the platform events on the fly calling the admin API.

Also propose if you think the platform event should be updated 

Sure I’ll share where I’m at :slightly_smiling_face: the idea is describe below in fact to possibly create event in trace OTLP format. Ideally it would be instrumentation in Flyte itself (in addition of the metrics). It’s just an exploration for now. Something like <https://openlineage.io/> might be a valuable too (but for somehting else) and probably easier to integrate with

Some folks are working on datahub integration

hey <@U039Q4WFL0J> - you were able to get all the ingress stuff sorted out I hope

and yeah, i started (barely) a doc for what we were hoping to achieve with datahub.  <@UTU7ZNA56> is planning on filling that out i believe.  the doc is <https://docs.google.com/document/d/1YMZkJr77Kg0IIJssFxs-QFMf2mpSEHIIP0hTVokqywY/edit#|here>

i filled in some of the background in that section in the bottom - feel free to take a look at that.

the reason this is relevant is because while we recognized a long time ago that the raw events that flyte emits were not great, we didn’t have a good use-case to inform us as to what proper events should look like.

datahub is one such use-case and yours sounds like it would help guide us as well.

the format we picked for now is just our own json schema.  one of the goals of a proper user-consumable events is that relevant information to the task/workflow/execution/etc would be included.  not sure we would be abusing oltp to include all of that.  definitely open to change though.  we didn’t pick json schema via any rigourous requirements, it just seemed like the easiest at the time.

Sounds great :slightly_smiling_face: The reason I mention openlineage is because their attempt is to have generic defintion for events around lineage that could integrate with different data catalog tools like Apache Atlas, Marquez or recently MSFT Preview but they also have integration with other framework like DBT or Spark. It seems that Datahub does yet support it but there is a <https://feature-requests.datahubproject.io/p/openlineage|FR> about it.
In the other hand when I mention OTLP it’s mainly to supplement the metrics and Log from Flyte using Tracing and Trace/Metric in context to integrate with most Observability tools and it follows <https://github.com/open-telemetry/opentelemetry-proto/tree/main/opentelemetry/proto/trace/v1|this> kind of schema