We started working on adding a Flyte source extens...
# announcementst
Tim Bauer
04/28/2022, 2:42 PMWe started working on adding a Flyte source extension to Datahub for tracking metadata and lineage about our Workflows. Is anyone else maybe already working on something similar?
👀 2
h
Haytham Abuelfutuh
04/28/2022, 2:43 PM
@Yee ^
Haytham Abuelfutuh
04/28/2022, 2:43 PM
@Rowan Shulver ^
h
Han
04/28/2022, 3:50 PMHi Tim. I spoke with Fredrik regarding this as well. Please keep me posted.
k
Ketan (kumare3)
04/29/2022, 2:33 PM
We actually have a prototype for this which we can share
🔥 1
Ketan (kumare3)
04/29/2022, 2:34 PM
We want to make this into an open source thing and ideally a framework for folks to extend
t
Tim Bauer
04/29/2022, 5:05 PMIt would be awesome if you could share that with us. My approach now was to write a new Datahub Source that makes use of flytekits “friendly” Client to pull data. Would be interesting to see which way you would have taken and rather focus on that if you prefer it.
k
Ketan (kumare3)
04/29/2022, 11:09 PM
Ohh this was to push data to datahub
Ketan (kumare3)
04/29/2022, 11:09 PM
We will share on Monday
Ketan (kumare3)
05/03/2022, 4:52 AM
@Tim Bauer here is a prototype for datahub - https://github.com/unionai/flyteevents-datahub
Ketan (kumare3)
05/03/2022, 4:53 AM
again this is not fully ready, but can be drastically improved
Ketan (kumare3)
05/03/2022, 4:53 AM
we probably should add a new plugin to flytekit to allow adding these sinks, and probably ship with datahub
167 Views