https://flyte.org
Join Slack
Hello there, I would like to get events equivalent to what is generated by `flytectl get execution ...
# announcements
m
Hello there, I would like to get events equivalent to what is generated by 
flytectl get execution -p flytesnacks -d development -o json
in the logs or somewhere for post-processing is there a way to do that? (I already have External events configured and processed by a lambda but I missing some context from it)
t
cc: @freezing-airport-6809
f
What do you mean - you can use the api
You want the output of flytectl
Flytectl uses the client
m
Yep, but I wanted to stream or react on those events but you’re right I can enrich the platform events on the fly calling the admin API.
f
Ya that’s what will be needed
Also propose if you think the platform event should be updated
m
Sure I’ll share where I’m at 🙂 the idea is describe below in fact to possibly create event in trace OTLP format. Ideally it would be instrumentation in Flyte itself (in addition of the metrics). It’s just an exploration for now. Something like https://openlineage.io/ might be a valuable too (but for somehting else) and probably easier to integrate with
f
Cc @thankful-minister-83577
Some folks are working on datahub integration
Cc @adamant-sandwich-39749 @worried-restaurant-93221
t
hey @mammoth-church-8767 - you were able to get all the ingress stuff sorted out I hope
👌 1
and yeah, i started (barely) a doc for what we were hoping to achieve with datahub. @colossal-painter-70298 is planning on filling that out i believe. the doc is here
👋 1
i filled in some of the background in that section in the bottom - feel free to take a look at that.
the reason this is relevant is because while we recognized a long time ago that the raw events that flyte emits were not great, we didn’t have a good use-case to inform us as to what proper events should look like.
datahub is one such use-case and yours sounds like it would help guide us as well.
the format we picked for now is just our own json schema. one of the goals of a proper user-consumable events is that relevant information to the task/workflow/execution/etc would be included. not sure we would be abusing oltp to include all of that. definitely open to change though. we didn’t pick json schema via any rigourous requirements, it just seemed like the easiest at the time.
m
Sounds great 🙂 The reason I mention openlineage is because their attempt is to have generic defintion for events around lineage that could integrate with different data catalog tools like Apache Atlas, Marquez or recently MSFT Preview but they also have integration with other framework like DBT or Spark. It seems that Datahub does yet support it but there is a FR about it. In the other hand when I mention OTLP it’s mainly to supplement the metrics and Log from Flyte using Tracing and Trace/Metric in context to integrate with most Observability tools and it follows this kind of schema
167 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by