this seems potentially really cool but i don t see an explai

Question

this seems potentially really cool but i don t see an explainer in the docs what does this do ```Annotated StructuredDataset subset cols ```

Yee · Answer

um

Yee · Answer

so like it s not entirely implemented or at least not efficiently but we have plans to round that out

Yee · Answer

the idea is that if you have a file sitting in s3 and it has 100 columns but you only need one column that allows you to just pull one column

Yee · Answer

but it s currently not fully implemented so it still pulls down the entire file but then deletes the ones you don t want

Yee · Answer

things like s3 select right

Yee · Answer

where did you get that from <https github com flyteorg flytesnacks blob master cookbook core type system structured dataset py|this>

Yee · Answer

we need to improve that documentation

Dylan Wilder · Answer

no <https docs flyte org projects cookbook en latest auto core type system structured dataset html|the docs here>

Dylan Wilder · Answer

couldn t that be done by just having a separate structured dataset with the subcols defined

Yee · Answer

what do you mean

Yee · Answer

yes for sure but having a separate structured dataset with the subcols defined is precisely what that is supposed to do