We started working on adding a Flyte source extension to

Question

We started working on adding a Flyte source extension to <https datahubproject io |Datahub> for tracking metadata and lineage about our Workflows Is anyone else maybe already working on something similar

Haytham Abuelfutuh · Answer

< Yee> ^

Haytham Abuelfutuh · Answer

< Rowan Shulver> ^

Han · Answer

Hi Tim I spoke with Fredrik regarding this as well Please keep me posted

Ketan (kumare3) · Answer

We actually have a prototype for this which we can share

Ketan (kumare3) · Answer

We want to make this into an open source thing and ideally a framework for folks to extend

Tim Bauer · Answer

It would be awesome if you could share that with us My approach now was to write a new Datahub Source that makes use of flytekits friendly Client to pull data Would be interesting to see which way you would have taken and rather focus on that if you prefer it

Ketan (kumare3) · Answer

Ohh this was to push data to datahub

Ketan (kumare3) · Answer

We will share on Monday

Ketan (kumare3) · Answer

< Tim Bauer> here is a prototype for datahub

Ketan (kumare3) · Answer

again this is not fully ready but can be drastically improved

Ketan (kumare3) · Answer

we probably should add a new plugin to flytekit to allow adding these sinks and probably ship with datahub