After a great call with <@UTU7ZNA56>, we stumbled ...
# datahub-flytev
Victor Gustavo da Silva Oliveira
02/16/2023, 8:24 PMAfter a great call with @Fredrik, we stumbled in some situations, that I believe we can think how to solve together. Some of them: How and where Flyte send its callback functions? Is it possible to send a callback in OpenLineage format? How can we capture it on DataHub's side? Fredrik tried using a REST call, but we believe that's not the best way to do it...
There where some others considerations, that Fredrik can help me to remember
From there, I think we can open a new issue on Flyte's repository, is that right?
y
Yee
02/16/2023, 10:01 PM
can we sync on a call some time?
Yee
02/16/2023, 10:01 PM
would early next week work for you? what tz are you in?
f
Fredrik
02/17/2023, 4:15 AMOpenlineage might not be the best format, as it seems to expect the whole package of metadata in the one and same JSON blob.This is not very flexible as you need to collect all info before you can send it onward. This in comparison with datahub’s MCP format which allows you to send incremental changes as they are discovered.
Fredrik
02/17/2023, 4:18 AMAnd the REST calls were made from my toy ”translator” service using the datahub sdk, but i could trivially have used Kafka as well. Datahub only supports REST and Kafka as methods to transfer data into datahub
Fredrik
02/17/2023, 4:19 AMThanks @Yee if you can help bring some clarity! 🙏
v
Victor Gustavo da Silva Oliveira
02/17/2023, 12:19 PMThanks for the help, @Fredrik!
@Yee next week works for me, I'm in GMT -3 timezone (Brasilia-DF timezone)
k
Ketan (kumare3)
02/22/2023, 6:33 AM
cc @Yee did we talk here?
y
Yee
02/22/2023, 5:32 PM
i have not. @Victor Gustavo da Silva Oliveira would you be open to a chat this week or next?
v
Victor Gustavo da Silva Oliveira
02/23/2023, 12:38 PMHello guys, sorry the delay, I'm moving to a new house, and things are a little messy right now. I can talk, sure! We just need to sync our agendas.
Victor Gustavo da Silva Oliveira
02/27/2023, 1:09 PM@Yee have you free this week for our talk? How about @Fredrik and @Ketan (kumare3)? Can we talk this week? I'll help in this project as a personal contribution, it will be interesting if our chat can be scheduled after my working hours, 7pm or later in my timezone. I believe it's around 3pm in Yee's timezone.
f
Fredrik
02/27/2023, 1:25 PMIt might be difficult to get everyones timezone to match, but you can invite me as optional
y
Yee
02/27/2023, 3:54 PM
yeah that’s fine. i can do weds this week? 3pm seattle time?
v
Victor Gustavo da Silva Oliveira
02/27/2023, 4:47 PMCool, this work for me. If you want to sync agendas, my email is victor.gso001@gmail.com
Victor Gustavo da Silva Oliveira
03/01/2023, 10:51 PM@Yee will you be able to talk today?
y
Yee
03/01/2023, 11:17 PM
oh crap sorry
Yee
03/01/2023, 11:17 PM
completely forgot
Yee
03/01/2023, 11:17 PM
are you still around?
v
Victor Gustavo da Silva Oliveira
03/02/2023, 12:06 PMSorry man, had to leave...
How about today today 11am Seattle time?
y
Yee
03/02/2023, 5:21 PM
sure.
10 Views