Hello all, @Victor Gustavo da Silva Oliveira here is interested in Datahub and Flyte integration. In the past - @Stephen and @Fredrik have also worked on it - there are few other folks in the community. I have also added folks from LinkedIn, who might also be interested (do not know yet). @Yee from my team was working with some folks, but he is out for the past 2 weeks and next 2 weeks. But, eventually he can help a lot. There is a sample that we created sometime ago, with some community folks - https://github.com/unionai/flyteevents-datahub. This is a prototype. It uses Flyte Events egress and replicates them to Datahub. You have to run the `

flytelineage

script Problems • This is not resilient, in case of failures does not do well. • We would ideally love this to be a framework - that folks can add new plugins if needed. For example Amundsen (spotify has an internal catalog) etc • Currently it simply listens to all events and keeps them in memory for a workflow and then replicates it. This is not needed, it could use flyte remote to get all the data only on receiving a terminal event. • No one uses this in production

We are not datahub experts, i think there is a channel in datahub as well to integrate with flyte

We chatted a lot about this with @Yee last fall, and got to a point where I would have needed to start experimenting with implementing some rudimentary support for emitting different types of events from Flyte. Life however got in the way and I have not been able the work on this since. The idea we were discussing was indeed to emit event from Flyte that then would be translated into Datahub/Amundsen/OpenLineage/etc events by a catalog-specific translator service. The event needs have been mapped out (to some extent) in this document, but it might be quite a tall order for me to start implementing these in Flyte in my current situation. Any help on this front would be highly appreciated. I’d be happy to work on a reference event translator (for datahub), though!

After a great call with @Fredrik, we stumbled in some situations, that I believe we can think how to solve together. Some of them: How and where Flyte send its callback functions? Is it possible to send a callback in OpenLineage format? How can we capture it on DataHub's side? Fredrik tried using a REST call, but we believe that's not the best way to do it... There where some others considerations, that Fredrik can help me to remember From there, I think we can open a new issue on Flyte's repository, is that right?

I had a chat with @Yee last week, and we talked about sync hour agendas to see if we can talk this week. @Fredrik when will you be available to talk? Let's try to do at a time that Yee can participate as well