this seems potentially really cool, but i don't se...
# flytekit
d
this seems potentially really cool, but i don't see an explainer in the docs. what does this do?
Copy code
Annotated[StructuredDataset, subset_cols]
y
um
so like… it’s not entirely implemented, or at least not efficiently, but we have plans to round that out
the idea is that if you have a file sitting in s3, and it has 100 columns, but you only need one column… that allows you to just pull one column
but it’s currently not fully implemented… so it still pulls down the entire file, but then deletes the ones you don’t want.
things like s3 select right?
where did you get that? from this?
we need to improve that documentation
d
couldn't that be done by just having a separate structured dataset with the subcols defined?
y
what do you mean?
yes for sure, but having a separate structured dataset with the subcols defined is precisely what that is supposed to do
111 Views