We started working on adding a Flyte source extens...
# announcements
We started working on adding a Flyte source extension to Datahub for tracking metadata and lineage about our Workflows. Is anyone else maybe already working on something similar?
👀 2
@Yee ^
@Rowan Shulver ^
Hi Tim. I spoke with Fredrik regarding this as well. Please keep me posted.
We actually have a prototype for this which we can share
🔥 1
We want to make this into an open source thing and ideally a framework for folks to extend
It would be awesome if you could share that with us. My approach now was to write a new Datahub Source that makes use of flytekits “friendly” Client to pull data. Would be interesting to see which way you would have taken and rather focus on that if you prefer it.
Ohh this was to push data to datahub
We will share on Monday
@Tim Bauer here is a prototype for datahub - https://github.com/unionai/flyteevents-datahub
again this is not fully ready, but can be drastically improved
we probably should add a new plugin to flytekit to allow adding these sinks, and probably ship with datahub