this seems potentially really cool, but i don't se...
# flytekitf
famous-businessperson-24711
08/03/2022, 4:03 PMthis seems potentially really cool, but i don't see an explainer in the docs. what does this do?
Copy code
Annotated[StructuredDataset, subset_cols]t
thankful-minister-83577
08/03/2022, 4:55 PM
um
thankful-minister-83577
08/03/2022, 4:56 PM
so like… it’s not entirely implemented, or at least not efficiently, but we have plans to round that out
thankful-minister-83577
08/03/2022, 4:57 PM
the idea is that if you have a file sitting in s3, and it has 100 columns, but you only need one column… that allows you to just pull one column
thankful-minister-83577
08/03/2022, 4:57 PM
but it’s currently not fully implemented… so it still pulls down the entire file, but then deletes the ones you don’t want.
thankful-minister-83577
08/03/2022, 5:01 PM
things like s3 select right?
thankful-minister-83577
08/03/2022, 5:03 PM
where did you get that? from this?
thankful-minister-83577
08/03/2022, 5:03 PM
we need to improve that documentation
f
famous-businessperson-24711
08/04/2022, 4:03 PMno, the docs here
famous-businessperson-24711
08/04/2022, 4:03 PMcouldn't that be done by just having a separate structured dataset with the subcols defined?
t
thankful-minister-83577
08/04/2022, 4:11 PM
what do you mean?
thankful-minister-83577
08/04/2022, 4:12 PM
yes for sure, but having a separate structured dataset with the subcols defined is precisely what that is supposed to do
161 Views