We started working on adding a Flyte source extens...
# announcementsw
worried-restaurant-93221
04/28/2022, 2:42 PMWe started working on adding a Flyte source extension to Datahub for tracking metadata and lineage about our Workflows. Is anyone else maybe already working on something similar?
👀 2
h
high-park-82026
04/28/2022, 2:43 PM
@thankful-minister-83577 ^
high-park-82026
04/28/2022, 2:43 PM
@bumpy-portugal-17701 ^
a
aloof-artist-49685
04/28/2022, 3:50 PMHi Tim. I spoke with Fredrik regarding this as well. Please keep me posted.
f
freezing-airport-6809
04/29/2022, 2:33 PM
We actually have a prototype for this which we can share
🔥 1
freezing-airport-6809
04/29/2022, 2:34 PM
We want to make this into an open source thing and ideally a framework for folks to extend
w
worried-restaurant-93221
04/29/2022, 5:05 PMIt would be awesome if you could share that with us. My approach now was to write a new Datahub Source that makes use of flytekits “friendly” Client to pull data. Would be interesting to see which way you would have taken and rather focus on that if you prefer it.
f
freezing-airport-6809
04/29/2022, 11:09 PM
Ohh this was to push data to datahub
freezing-airport-6809
04/29/2022, 11:09 PM
We will share on Monday
freezing-airport-6809
05/03/2022, 4:52 AM
@worried-restaurant-93221 here is a prototype for datahub - https://github.com/unionai/flyteevents-datahub
freezing-airport-6809
05/03/2022, 4:53 AM
again this is not fully ready, but can be drastically improved
freezing-airport-6809
05/03/2022, 4:53 AM
we probably should add a new plugin to flytekit to allow adding these sinks, and probably ship with datahub
173 Views