https://flyte.org logo
#announcements
Title
# announcements
j

Jonathan Lamiel

06/24/2022, 9:23 AM
Hello there, I would like to get events equivalent to what is generated by
flytectl get execution -p flytesnacks -d development -o json
in the logs or somewhere for post-processing is there a way to do that? (I already have External events configured and processed by a lambda but I missing some context from it)
s

Samhita Alla

06/24/2022, 1:50 PM
cc: @Ketan (kumare3)
k

Ketan (kumare3)

06/24/2022, 2:12 PM
What do you mean - you can use the api
You want the output of flytectl
Flytectl uses the client
j

Jonathan Lamiel

06/24/2022, 2:27 PM
Yep, but I wanted to stream or react on those events but you’re right I can enrich the platform events on the fly calling the admin API.
k

Ketan (kumare3)

06/24/2022, 2:51 PM
Ya that’s what will be needed
Also propose if you think the platform event should be updated
j

Jonathan Lamiel

06/24/2022, 3:11 PM
Sure I’ll share where I’m at 🙂 the idea is describe below in fact to possibly create event in trace OTLP format. Ideally it would be instrumentation in Flyte itself (in addition of the metrics). It’s just an exploration for now. Something like https://openlineage.io/ might be a valuable too (but for somehting else) and probably easier to integrate with
k

Ketan (kumare3)

06/24/2022, 5:10 PM
Cc @Yee
Some folks are working on datahub integration
Cc @Fredrick @Tim Bauer
y

Yee

06/24/2022, 6:17 PM
hey @Jonathan Lamiel - you were able to get all the ingress stuff sorted out I hope
👌 1
and yeah, i started (barely) a doc for what we were hoping to achieve with datahub. @Fredrik is planning on filling that out i believe. the doc is here
👋 1
i filled in some of the background in that section in the bottom - feel free to take a look at that.
the reason this is relevant is because while we recognized a long time ago that the raw events that flyte emits were not great, we didn’t have a good use-case to inform us as to what proper events should look like.
datahub is one such use-case and yours sounds like it would help guide us as well.
the format we picked for now is just our own json schema. one of the goals of a proper user-consumable events is that relevant information to the task/workflow/execution/etc would be included. not sure we would be abusing oltp to include all of that. definitely open to change though. we didn’t pick json schema via any rigourous requirements, it just seemed like the easiest at the time.
j

Jonathan Lamiel

06/27/2022, 9:18 AM
Sounds great 🙂 The reason I mention openlineage is because their attempt is to have generic defintion for events around lineage that could integrate with different data catalog tools like Apache Atlas, Marquez or recently MSFT Preview but they also have integration with other framework like DBT or Spark. It seems that Datahub does yet support it but there is a FR about it. In the other hand when I mention OTLP it’s mainly to supplement the metrics and Log from Flyte using Tracing and Trace/Metric in context to integrate with most Observability tools and it follows this kind of schema
76 Views