After a great call with <@UTU7ZNA56>, we stumbled ...
# datahub-flyted
dry-ability-69144
02/16/2023, 8:24 PMAfter a great call with @colossal-painter-70298, we stumbled in some situations, that I believe we can think how to solve together. Some of them: How and where Flyte send its callback functions? Is it possible to send a callback in OpenLineage format? How can we capture it on DataHub's side? Fredrik tried using a REST call, but we believe that's not the best way to do it...
There where some others considerations, that Fredrik can help me to remember
From there, I think we can open a new issue on Flyte's repository, is that right?
t
thankful-minister-83577
02/16/2023, 10:01 PM
can we sync on a call some time?
thankful-minister-83577
02/16/2023, 10:01 PM
would early next week work for you? what tz are you in?
c
colossal-painter-70298
02/17/2023, 4:15 AMOpenlineage might not be the best format, as it seems to expect the whole package of metadata in the one and same JSON blob.This is not very flexible as you need to collect all info before you can send it onward. This in comparison with datahub’s MCP format which allows you to send incremental changes as they are discovered.
👀 1
🎉 1
colossal-painter-70298
02/17/2023, 4:18 AMAnd the REST calls were made from my toy ”translator” service using the datahub sdk, but i could trivially have used Kafka as well. Datahub only supports REST and Kafka as methods to transfer data into datahub
👍 1
colossal-painter-70298
02/17/2023, 4:19 AMThanks @thankful-minister-83577 if you can help bring some clarity! 🙏
d
dry-ability-69144
02/17/2023, 12:19 PMThanks for the help, @colossal-painter-70298!
@thankful-minister-83577 next week works for me, I'm in GMT -3 timezone (Brasilia-DF timezone)
f
freezing-airport-6809
02/22/2023, 6:33 AM
cc @thankful-minister-83577 did we talk here?
t
thankful-minister-83577
02/22/2023, 5:32 PM
i have not. @dry-ability-69144 would you be open to a chat this week or next?
d
dry-ability-69144
02/23/2023, 12:38 PMHello guys, sorry the delay, I'm moving to a new house, and things are a little messy right now. I can talk, sure! We just need to sync our agendas.
dry-ability-69144
02/27/2023, 1:09 PM@thankful-minister-83577 have you free this week for our talk? How about @colossal-painter-70298 and @freezing-airport-6809? Can we talk this week? I'll help in this project as a personal contribution, it will be interesting if our chat can be scheduled after my working hours, 7pm or later in my timezone. I believe it's around 3pm in Yee's timezone.
c
colossal-painter-70298
02/27/2023, 1:25 PMIt might be difficult to get everyones timezone to match, but you can invite me as optional
t
thankful-minister-83577
02/27/2023, 3:54 PM
yeah that’s fine. i can do weds this week? 3pm seattle time?
d
dry-ability-69144
02/27/2023, 4:47 PMCool, this work for me. If you want to sync agendas, my email is victor.gso001@gmail.com
dry-ability-69144
03/01/2023, 10:51 PM@thankful-minister-83577 will you be able to talk today?
t
thankful-minister-83577
03/01/2023, 11:17 PM
oh crap sorry
thankful-minister-83577
03/01/2023, 11:17 PM
completely forgot
thankful-minister-83577
03/01/2023, 11:17 PM
are you still around?
d
dry-ability-69144
03/02/2023, 12:06 PMSorry man, had to leave...
How about today today 11am Seattle time?
t
thankful-minister-83577
03/02/2023, 5:21 PM
sure.
13 Views