hey flyte team, we finally got around to investigating migration to structured dataset! We have a few questions about the changes, maybe it makes sense to hop on a call? We're also open to submitting PRs if things make sense cc @thankful-minister-83577 (who's on vaca 🙂 ) @mammoth-lizard-41416
• There are some api structural changes, we're curious if they're intentional (eg requiring Annotated, loss of
open().write()
)
• We have an issue setting the bigquery project for perms for bq URIs, there's a couple approaches to fixing this we can consider
• There doesn't appear to be any type checking or casting of types based on the schema. this could be pretty valuable
t
thankful-minister-83577
08/19/2022, 4:50 PM
nah let’s set up a quick chat.
thankful-minister-83577
08/19/2022, 4:50 PM
i can go over the thought process. also writing a blog on this soon so would like to make sure we have it right.
f
famous-businessperson-24711
08/19/2022, 5:56 PM
cool. monday?
t
thankful-minister-83577
08/19/2022, 9:16 PM
for anyone else paying attention, we had a brief chat about this. One point to investigate that came up was around type-casting and strictness. https://github.com/flyteorg/flyte/issues/2794
thankful-minister-83577
08/19/2022, 9:17 PM
Also https://github.com/flyteorg/flyte/issues/2795 has been made for moving the bigquery writer to use the lower level library. Dylan mind adding any code snippets you have to that ticket too please?
thankful-minister-83577
08/19/2022, 9:17 PM
thanks!
thankful-minister-83577
08/19/2022, 9:25 PM
actually dylan or @mammoth-lizard-41416 would you guys be up for contributing that change? the bigquery one.