Hi all seems the tutorials are all using outdated flytekit v

Question

Hi all seems the tutorials are all using outdated flytekit versions trying to update the pima diabetes Any examples on how to transform `FlyteSchema FEATURE COLUMNS ` to `StructuredDataset` ```FEATURE COLUMNS = OrderedDict k v for k v in DATASET COLUMNS items if k = class def split traintest dataset dataset FlyteFile typing TypeVar csv seed int test split ratio float gt Tuple FlyteSchema FEATURE COLUMNS FlyteSchema FEATURE COLUMNS FlyteSchema CLASSES COLUMNS FlyteSchema CLASSES COLUMNS ```

Kevin Su · Answer

```FlyteSchema FEATURE COLUMNS ``` update to ```StructuredDataset FEATURE COLUMNS ```

Kevin Su · Answer

you re right we should remove flyte schema from the docs

Hud · Answer

< Kevin Su> yeah I tried that but got this ```TypeError type object is not subscriptable``` Is this on flytekit 1 8 1

Kevin Su · Answer

let me test it on my side I ll get back to you

Hud · Answer

< Kevin Su> I replaced the `FlyteSchema FEATURE COLUMNS ` gt `StructuredDataset` But got the following error ```TypeError Encountered error while executing workflow diabetes xgboost model Error encountered while executing diabetes xgboost model Failed to convert outputs of task split traintest dataset at position 0 Failed to find a handler for lt class flytekit types structured structured dataset StructuredDataset gt protocol file fmt ```

Hud · Answer

Not sure if related but getting the same error from this <https docs flyte org projects cookbook en latest auto examples type system structured dataset html|guide> too ```Failed with Unknown Exception lt class ValueError gt Reason Encountered error while executing workflow structured dataset schema compatibility wf Error encountered while executing schema compatibility wf Failed to find a handler for lt class numpy ndarray gt protocol file fmt parquet Encountered error while executing workflow structured dataset schema compatibility wf Error encountered while executing schema compatibility wf Failed to find a handler for lt class numpy ndarray gt protocol file fmt parquet ```

Kevin Su · Answer

looking testing now

Kevin Su · Answer

just ran the example in this <https docs flyte org projects cookbook en latest auto examples type system structured dataset html|doc> it works for me could share the workflow code you ran

Kevin Su · Answer

gt I replaced the `FlyteSchema FEATURE COLUMNS ` gt `StructuredDataset` gt But got the following error could you try ```Annotated StructuredDataset FEATURE COLUMNS ```

Hud · Answer

Hm that s odd so this was the code for `structured dataset py` ```import os import typing import numpy as np import pandas as pd import pyarrow as pa import pyarrow parquet as pq from flytekit import FlyteContext StructuredDatasetType kwtypes task workflow from flytekit models import literals from flytekit models literals import StructuredDatasetMetadata from flytekit types schema import FlyteSchema from flytekit types structured structured dataset import PARQUET StructuredDataset StructuredDatasetDecoder StructuredDatasetEncoder StructuredDatasetTransformerEngine from typing extensions import Annotated superset cols = kwtypes Name=str Age=int Height=int subset cols = kwtypes Age=int task def get df a int gt Annotated pd DataFrame superset cols Generate a sample dataframe return pd DataFrame Name Tom Joseph Age a 22 Height 160 178 task def get schema df a int gt FlyteSchema superset cols Generate a sample dataframe s = FlyteSchema s open write pd DataFrame Name Tom Joseph Age a 22 Height 160 178 return s task def get subset df df Annotated StructuredDataset subset cols gt Annotated StructuredDataset subset cols df = df open pd DataFrame all df = pd concat df pd DataFrame 30 columns= Age On specifying BigQuery uri for StructuredDataset flytekit writes a pandas dataframe to a BigQuery table return StructuredDataset dataframe=df task def to numpy ds Annotated StructuredDataset subset cols gt Annotated StructuredDataset subset cols PARQUET numpy array = ds open np ndarray all return StructuredDataset dataframe=numpy array workflow def pandas compatibility wf a int gt Annotated StructuredDataset subset cols df = get df a=a ds = get subset df df=df noqa shown for demonstration users should use the same types between tasks return to numpy ds=ds workflow def schema compatibility wf a int gt Annotated StructuredDataset subset cols df = get schema df a=a ds = get subset df df=df noqa shown for demonstration users should use the same types between tasks return to numpy ds=ds ``` And ran it locally `pyflyte run structured dataset py schema compatibility wf a 42`

Kevin Su · Answer

run into the same error looking

Kevin Su · Answer

you need to register numpy encoder and decoder <https docs flyte org projects cookbook en latest auto examples type system structured dataset html numpy encoder and decoder>

Hud · Answer

Ah I think it runs So I d have to register the numpy encoder and decoder everytime I guess I ll run this on `pima diabetes` and get back to you

Hud · Answer

gt so I made some changes to `pima diabetes` Basically updating `FlyteSchema FEATURE COLUMNS ` gt `Annotated SructuredDataset FEATURE COLUMNS ` ``` following the numpy encoder decoder above task def get feature struct data df Annotated StructuredDataset FEATURE COLUMNS gt Annotated StructuredDataset FEATURE COLUMNS df = df open pd DataFrame all return StructuredDataset dataframe=df task def features to numpy ds Annotated StructuredDataset FEATURE COLUMNS gt Annotated StructuredDataset FEATURE COLUMNS PARQUET numpy array = ds open np ndarray all return StructuredDataset dataframe=numpy array task def transform feature schema df Annotated StructuredDataset FEATURE COLUMNS gt Annotated StructuredDataset FEATURE COLUMNS ds = get feature struct data df=df return features to numpy ds=ds task cache version= 1 0 cache=True limits=Resources mem= 200Mi def split traintest dataset dataset FlyteFile typing TypeVar csv seed int test split ratio float gt Tuple Annotated StructuredDataset FEATURE COLUMNS Annotated StructuredDataset FEATURE COLUMNS Annotated StructuredDataset CLASSES COLUMNS Annotated StructuredDataset CLASSES COLUMNS column names = k for k in DATASET COLUMNS keys df = pd read csv dataset names=column names Select all features x = df column names 8 Select only the classes y = df column names 1 split data into train and test sets x train df x test df y train df y test df = train test split x y test size=test split ratio random state=seed return transform feature schema df=x train df transform feature schema df=x test df transform classes schema df=y train df CLASSES COLUMNS instead of FEATURE COLUMNS transform classes schema df=y test df task cache version= 1 0 cache=True limits=Resources mem= 200Mi def predict x Annotated StructuredDataset FEATURE COLUMNS model ser FlyteFile MODELSER JOBLIB gt Annotated StructuredDataset CLASSES COLUMNS model = joblib load model ser x df = x open pd DataFrame all y pred = model predict x df col = k for k in CLASSES COLUMNS keys y pred df = pd DataFrame y pred columns=col dtype= int64 y pred df round 0 return y pred df``` got this error seemingly during `predictions = predict x=x test model ser=model model ` ```TypeError Encountered error while executing workflow diabetes xgboost model Error encountered while executing diabetes xgboost model Failed to convert outputs of task predict at position 0 Failed to find a handler for lt class flytekit types structured structured dataset StructuredDataset gt protocol file fmt ```

Samhita Alla · Answer

Can you convert `y pred df` to `StructuredDataset dataframe=y pred df `

Samhita Alla · Answer

Also calling tasks from within tasks essentially means you re calling functions within tasks If you want to call functions within your tasks you need not decorate them with the ` task` decorator Just keep them as Python functions You need to call your tasks from within a Flyte workflow

Hud · Answer

Sorry for the late reply so the `split traintest dataset` task works albeit without the numpy encoding decoding but simply following < Samhita Alla>s suggestion ``` task container image=image cache version= 1 0 cache=True limits=Resources mem= 200Mi def split traintest dataset dataset FlyteFile typing TypeVar csv seed int test split ratio float gt Tuple Annotated StructuredDataset FEATURE COLUMNS Annotated StructuredDataset FEATURE COLUMNS Annotated StructuredDataset CLASSES COLUMNS Annotated StructuredDataset CLASSES COLUMNS column names = k for k in DATASET COLUMNS keys df = pd read csv dataset names=column names x = df column names 8 y = df column names 1 x train df x test df y train df y test df = train test split x y test size=test split ratio random state=seed return StructuredDataset dataframe=x train df StructuredDataset dataframe=x test df StructuredDataset dataframe=y train df StructuredDataset dataframe=y test df ```

Hud · Answer

But now the `joblib dump` doesnt work in the sandbox ```Traceback most recent call last File usr local lib python3 9 site packages flytekit exceptions scopes py line 206 in user entry point return wrapped args kwargs File root pima diabetes diabetes py line 182 in fit joblib dump m fname File usr local lib python3 9 site packages joblib numpy pickle py line 552 in dump with open filename wb as f Message Errno 13 Permission denied model joblib dat User error ``` how do we deal with model dumps between tasks

Samhita Alla · Answer

What s `fname`

Samhita Alla · Answer

It needs to be `os path join flytekit current context working directory your file name `

Samhita Alla · Answer

Thank you for your willingness to contribute to Flyte I ve actually opened an issue to include ImageSpec in all our examples <https github com flyteorg flyte issues 3790> I ll be working on it very soon But please feel free to create PRs to update the documentation as you deem appropriate A PR for every example should be fine or if you prefer you can create a PR that encompasses multiple examples

Hud · Answer

Ok it seems a standardized `ImageSpec` definition is yet to be decided I ll link any working PRs to that issue then Also seems defining `List Annotated StructuredDataset lt SomeOrderedDict gt ` gives the `TypeError unhashable type dict ` how to handle a list of structured datasets

Samhita Alla · Answer

I don t understand What s `dict` in the type annotation you specified apart from ordered dict

Hud · Answer

so in `house price prediction multiregion house price predictor py` around L76 ```dataset = typing NamedTuple GenerateSplitDataOutputs train data=typing List StructuredDataset DATASET COLUMNS previously `typing List pd DataFrame ` val data=typing List StructuredDataset DATASET COLUMNS test data=typing List StructuredDataset DATASET COLUMNS DATASET COLUMNS = PRICE int YEAR BUILT int SQUARE FEET int NUM BEDROOMS int NUM BATHROOMS float LOT ACRES float GARAGE SPACES int ``` It isn t clear from the full error ``` Users hud venvs house price prediction bin python Users hud projects flytesnacks examples house price prediction house price prediction multiregion house price predictor py Traceback most recent call last Users hud projects flytesnacks examples house price prediction house price prediction multiregion house price predictor py 90 in lt module gt 90 def generate and split data multiloc Users hud venvs house price prediction lib python3 9 site packages flytekit core task py 260 in wrapper 260 task instance = TaskPlugins find pythontask plugin type task config Users hud venvs house price prediction lib python3 9 site packages flytekit core tracker py 35 in call 35 o = super InstanceTrackingMeta cls call args kwargs Users hud venvs house price prediction lib python3 9 site packages flytekit core python function task py 121 in init 121 super init Users hud venvs house price prediction lib python3 9 site packages flytekit core python auto container py 85 in init 85 super init Users hud venvs house price prediction lib python3 9 site packages flytekit core base task py 420 in init 420 interface=transform interface to typed interface interface Users hud venvs house price prediction lib python3 9 site packages flytekit core interface py 248 in transform interface to typed interface 248 outputs map = transform variable map interface outputs output descriptions Users hud venvs house price prediction lib python3 9 site packages flytekit core interface py 345 in transform variable map 345 res k = transform type v descriptions get k k Users hud venvs house price prediction lib python3 9 site packages flytekit core interface py 350 in transform type 350 return interface models Variable type=TypeEngine to literal type x description=de Users hud venvs house price prediction lib python3 9 site packages flytekit core type engine py 826 in to literal type 826 transformer = cls get transformer python type Users hud venvs house price prediction lib python3 9 site packages flytekit core type engine py 747 in get transformer 747 if python type in cls REGISTRY Users hud pyenv versions 3 9 17 lib python3 9 typing py 757 in hash 757 return hash self origin self args Users hud pyenv versions 3 9 17 lib python3 9 typing py 1294 in hash 1294 return hash self origin self metadata TypeError unhashable type dict ``` But that error occurs when I append the `Annotated ` in a list

Samhita Alla · Answer

< Hud> would you mind letting me know why you are replacing dataframe with structured dataset

Hud · Answer

< Samhita Alla> Oh I guess just for the purpose of validating types defined in the `DATASET COLUMNS` Or for which use cases are Structured Datasets generally used for

Samhita Alla · Answer

In that case you can use `Annotated pd DataFrame lt your columns gt ` So are you telling `list Annotated pd DataFrame lt your columns gt ` isn t working for you

Hud · Answer

yes both `list Annotated StructuredDataset lt gt ` or `list Annotated pd DataFrame lt gt ` do not work for me whatever type I put in ` lt gt ` gives me unhashable error for that type

Samhita Alla · Answer

Can you create an issue please

Samhita Alla · Answer

flyte bug

Slackbot · Answer

ladybug Create a new Flyte Bug issue <https github com flyteorg flyte issues new assignees= amp labels=bug 2Cuntriaged amp template=bug report yaml amp title= 5BBUG 5D+>

Hud · Answer

Okay so using a dataclass solves the issue <https github com flyteorg flyte issues 3959>