this seems potentially really cool, but i don't se...
# flytekitd
Dylan Wilder
08/03/2022, 4:03 PMthis seems potentially really cool, but i don't see an explainer in the docs. what does this do?
Copy code
Annotated[StructuredDataset, subset_cols]y
Yee
08/03/2022, 4:55 PM
um
Yee
08/03/2022, 4:56 PM
so like… it’s not entirely implemented, or at least not efficiently, but we have plans to round that out
Yee
08/03/2022, 4:57 PM
the idea is that if you have a file sitting in s3, and it has 100 columns, but you only need one column… that allows you to just pull one column
Yee
08/03/2022, 4:57 PM
but it’s currently not fully implemented… so it still pulls down the entire file, but then deletes the ones you don’t want.
Yee
08/03/2022, 5:01 PM
things like s3 select right?
Yee
08/03/2022, 5:03 PM
where did you get that? from this?
Yee
08/03/2022, 5:03 PM
we need to improve that documentation
d
Dylan Wilder
08/04/2022, 4:03 PMno, the docs here
Dylan Wilder
08/04/2022, 4:03 PMcouldn't that be done by just having a separate structured dataset with the subcols defined?
y
Yee
08/04/2022, 4:11 PM
what do you mean?
Yee
08/04/2022, 4:12 PM
yes for sure, but having a separate structured dataset with the subcols defined is precisely what that is supposed to do
160 Views