Thread
#flytekit
    Dylan Wilder

    Dylan Wilder

    1 month ago
    this seems potentially really cool, but i don't see an explainer in the docs. what does this do?
    Annotated[StructuredDataset, subset_cols]
    Yee

    Yee

    1 month ago
    um
    so like… it’s not entirely implemented, or at least not efficiently, but we have plans to round that out
    the idea is that if you have a file sitting in s3, and it has 100 columns, but you only need one column… that allows you to just pull one column
    but it’s currently not fully implemented… so it still pulls down the entire file, but then deletes the ones you don’t want.
    things like s3 select right?
    where did you get that? from this?
    we need to improve that documentation
    Dylan Wilder

    Dylan Wilder

    1 month ago
    couldn't that be done by just having a separate structured dataset with the subcols defined?
    Yee

    Yee

    1 month ago
    what do you mean?
    yes for sure, but having a separate structured dataset with the subcols defined is precisely what that is supposed to do