Hello there I would like to get events equivalent to what is

Question

Hello there I would like to get events equivalent to what is generated by `flytectl get execution p flytesnacks d development o json` in the logs or somewhere for post processing is there a way to do that I already have External events configured and processed by a lambda but I missing some context from it

Samhita Alla · Answer

cc < Ketan kumare3 >

Ketan (kumare3) · Answer

What do you mean you can use the api

Ketan (kumare3) · Answer

You want the output of flytectl

Ketan (kumare3) · Answer

Flytectl uses the client

Jonathan Lamiel · Answer

Yep but I wanted to stream or react on those events but you re right I can enrich the platform events on the fly calling the admin API

Ketan (kumare3) · Answer

Ya that s what will be needed

Ketan (kumare3) · Answer

Also propose if you think the platform event should be updated

Jonathan Lamiel · Answer

Sure I ll share where I m at slightly smiling face the idea is describe below in fact to possibly create event in trace OTLP format Ideally it would be instrumentation in Flyte itself in addition of the metrics It s just an exploration for now Something like <https openlineage io > might be a valuable too but for somehting else and probably easier to integrate with

Ketan (kumare3) · Answer

Cc < Yee>

Ketan (kumare3) · Answer

Some folks are working on datahub integration

Ketan (kumare3) · Answer

Cc < Fredrick> < Tim Bauer>

Yee · Answer

hey < Jonathan Lamiel> you were able to get all the ingress stuff sorted out I hope

Yee · Answer

and yeah i started barely a doc for what we were hoping to achieve with datahub < Fredrik> is planning on filling that out i believe the doc is

Yee · Answer

i filled in some of the background in that section in the bottom feel free to take a look at that

Yee · Answer

the reason this is relevant is because while we recognized a long time ago that the raw events that flyte emits were not great we didn t have a good use case to inform us as to what proper events should look like

Yee · Answer

datahub is one such use case and yours sounds like it would help guide us as well

Yee · Answer

the format we picked for now is just our own json schema one of the goals of a proper user consumable events is that relevant information to the task workflow execution etc would be included not sure we would be abusing oltp to include all of that definitely open to change though we didn t pick json schema via any rigourous requirements it just seemed like the easiest at the time

Jonathan Lamiel · Answer

Sounds great slightly smiling face The reason I mention openlineage is because their attempt is to have generic defintion for events around lineage that could integrate with different data catalog tools like Apache Atlas Marquez or recently MSFT Preview but they also have integration with other framework like DBT or Spark It seems that Datahub does yet support it but there is a about it In the other hand when I mention OTLP it s mainly to supplement the metrics and Log from Flyte using Tracing and Trace Metric in context to integrate with most Observability tools and it follows kind of schema