Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-on-gcp
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
integrations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
show-and-tell
sig-large-models
torch-elastic
workflow-building-ui-proj
writing-w-sfloris
Title
v
Victor Gustavo da Silva Oliveira
02/16/2023, 8:24 PMAfter a great call with @Fredrik, we stumbled in some situations, that I believe we can think how to solve together. Some of them: How and where Flyte send its callback functions? Is it possible to send a callback in OpenLineage format? How can we capture it on DataHub's side? Fredrik tried using a REST call, but we believe that's not the best way to do it...
There where some others considerations, that Fredrik can help me to remember
From there, I think we can open a new issue on Flyte's repository, is that right?
y
Yee
02/16/2023, 10:01 PMcan we sync on a call some time?
would early next week work for you? what tz are you in?
f
Fredrik
02/17/2023, 4:15 AMOpenlineage might not be the best format, as it seems to expect the whole package of metadata in the one and same JSON blob.This is not very flexible as you need to collect all info before you can send it onward. This in comparison with datahub’s MCP format which allows you to send incremental changes as they are discovered.
And the REST calls were made from my toy ”translator” service using the datahub sdk, but i could trivially have used Kafka as well. Datahub only supports REST and Kafka as methods to transfer data into datahub
Thanks @Yee if you can help bring some clarity! 🙏
v
Victor Gustavo da Silva Oliveira
02/17/2023, 12:19 PMThanks for the help, @Fredrik!
@Yee next week works for me, I'm in GMT -3 timezone (Brasilia-DF timezone)
k
Ketan (kumare3)
02/22/2023, 6:33 AMcc @Yee did we talk here?
y
Yee
02/22/2023, 5:32 PMi have not. @Victor Gustavo da Silva Oliveira would you be open to a chat this week or next?
v
Victor Gustavo da Silva Oliveira
02/23/2023, 12:38 PMHello guys, sorry the delay, I'm moving to a new house, and things are a little messy right now. I can talk, sure! We just need to sync our agendas.
@Yee have you free this week for our talk? How about @Fredrik and @Ketan (kumare3)? Can we talk this week? I'll help in this project as a personal contribution, it will be interesting if our chat can be scheduled after my working hours, 7pm or later in my timezone. I believe it's around 3pm in Yee's timezone.
f
Fredrik
02/27/2023, 1:25 PMIt might be difficult to get everyones timezone to match, but you can invite me as optional
y
Yee
02/27/2023, 3:54 PMyeah that’s fine. i can do weds this week? 3pm seattle time?
v
Victor Gustavo da Silva Oliveira
02/27/2023, 4:47 PMCool, this work for me. If you want to sync agendas, my email is victor.gso001@gmail.com
@Yee will you be able to talk today?
y
Yee
03/01/2023, 11:17 PMoh crap sorry
completely forgot
are you still around?
v
Victor Gustavo da Silva Oliveira
03/02/2023, 12:06 PMSorry man, had to leave...
How about today today 11am Seattle time?
y
Yee
03/02/2023, 5:21 PMsure.