Hey everyone again raised hands I see a stuck in triggering

Question

Hey everyone again raised hands I see a stuck in triggering to the next task My main workflow is depicted in attached pic First `preproc and split` step was executed successfully ```pyflyte execute inputs output prefix raw output data prefix checkpoint path prev checkpoint resolver flytekit core python auto container default task resolver task module app workflow task name preproc and split``` On the output one should expect a small train dataset with 50k records In Nods allocation i see a sufficient mem available But then the first task has been succeeded i see an eternal hang in this step and flyte don t produce next executions according to the workflow `task resource defaults` conf is the next ``` task resource defaults yaml | task resources defaults cpu 1 memory 3000Mi storage 200Mi limits cpu 5 gpu 1 memory 8Gi storage 500Mi``` I have one task that generates a dataclasses instances on the exit and another task should takes these classes as input params ``` workflow def main flow gt Forecast Main Flyte WorkFlow consisting of three tasks preproc and split train xgboost clf get predictions log= START START Raw Preprocessing and Splitting timestamp=None train cls target cls = preproc and split log= START START Initialize Boosting Params timestamp=None saved mpath = train xgboost clf feat cls=train cls target cls=target cls xgb params=xgb params cust metric=BoostingCustMetric ``` Where ```def preproc and split gt Tuple Fraud Raw PostProc Data Class Fraud Raw Target Data Class ``` Any advices why I faced this behaviour

Ketan (kumare3) · Answer

Hi Anthony welcome to the community this is odd

Ketan (kumare3) · Answer

The thing is next node is not even queued

Ketan (kumare3) · Answer

< Shivay Lamba> can you run this example and let s get a reproduction seems odd

Shivay Lamba · Answer

Sure I will try to run this and share the result here

Anthony · Answer

i cannot share due to PII a private dataset but i wanna test flyte on some real cases not toy examples and here im downsampled ~50k records for training First `preproc and split` task has been executed successfully but I cannot pass its output to the next training task I have 14 Gb mem in my local machine in total `task resource defaults` conf is the next ``` task resource defaults yaml | task resources defaults cpu 1 memory 3000Mi storage 200Mi limits cpu 5 gpu 1 memory 8Gi storage 500Mi```

Yee · Answer

are all the tasks defined in the same file

Ketan (kumare3) · Answer

But shouldn t that simply fail to load task

Anthony · Answer

< Yee> Yes all task are defined within one script look at workflow py BTW I conduct the next experiments and got new odd behaviour Reinit flytectl sandbox Edit gt `kubectl n flyte edit cm flyte admin base config` Edit gt `kubectl edit configmap n flyte flyte propeller config` REMOVE task decorator for `preproc and split ` So only two task has been added to flow ``` task cache=False cache version=CACHE VERSION requests=request resources limits=limit resources def train xgboost clf feat cls Fraud Raw PostProc Data Class target cls Fraud Raw Target Data Class xgb params dict cust metric BoostingCustMetric gt JoblibSerializedFile ``` and next one ``` task cache=False cache version=CACHE VERSION requests=request resources limits=limit resources def get predictions feat cls Fraud Raw PostProc Data Class target cls Fraud Raw Target Data Class model path JoblibSerializedFile thresh float gt Forecast ``` rocket Workflow ``` workflow def main flow gt Forecast Main Flyte WorkFlow consisting of three tasks preproc and split train xgboost clf get predictions log= START START Raw Preprocessing and Splitting timestamp=None train cls target cls = preproc and split log= START START Initialize Boosting Params timestamp=None saved mpath = train xgboost clf feat cls=train cls target cls=target cls xgb params=xgb params cust metric=BoostingCustMetric forecast = get predictions feat cls=train cls target cls=target cls model path=saved mpath thresh=THRESH serialize model model=xgb clf return forecast``` Run workflow locally `pyflyte run app workflow py main flow` SUCCESS white check mark Ok going further to `serialise` and `register` ```pyflyte c flyte sandbox config pkgs app package force in container source path root image anti fraud ml train c20b6ee57601c0ddf12874fe671603b5ed78ece8``` `pyflyte` first runs my function `preproc and split` that produces ``` lt class pandas core frame DataFrame gt 53621 46 ``` Here is a final log output for serialising all pb files So we successfully serialised 4 flyte objects and packaged them ```2022 09 29 16 37 04 830|anti fraud ml train END Feature Transformation Pipeline 2022 09 29 16 37 04 830|anti fraud ml train START Train data info lt class pandas core frame DataFrame gt 53621 46 2022 09 29 16 37 04 879|anti fraud ml train START START Initialize Boosting Params asctime 2022 09 29 17 37 20 155 name flytekit levelname WARNING message Failed to extract schema for object lt class fraud preproc model dataclass Forecast gt will run schemaless error unsupported field type lt fields TimestampField dump default= lt marshmallow missing gt attribute=None validate=None required=False load only=False dump only=False load default= lt marshmallow missing gt allow none=False error messages= required Missing data for required field null Field may not be null validator failed Invalid value gt If you have postponed annotations turned on PEP 563 turn it off please Postponedevaluation doesn t work with json dataclasses Successfully serialized 4 flyte objects Packaging app workflow train xgboost clf gt 0 app workflow train xgboost clf 1 pb Packaging app workflow get predictions gt 1 app workflow get predictions 1 pb Packaging app workflow main flow gt 2 app workflow main flow 2 pb Packaging app workflow main flow gt 3 app workflow main flow 3 pb Successfully packaged 4 flyte objects into Users aektov pyflyte anti fraud ml train flyte package tgz``` Lets register all objects ```flytectl c flyte sandbox config yml register files project flyte anti fraud ml domain development archive flyte package tgz outputLocationPrefix force version c20b6ee57601c0ddf12874fe671603b5ed78ece8``` Oops we got a error ``` | NAME | STATUS | ADDITIONAL INFO | | tmp register2016944902 0 app workflow train xgboost clf 1 pb | Success | AlreadyExists | | tmp register2016944902 1 app workflow get predictions 1 pb | Success | AlreadyExists | | tmp register2016944902 2 app workflow main flow 2 pb | Failed | Error registering file due to rpc error code = | | | | ResourceExhausted desc = grpc received message larger | | | | than max 57552210 vs 4194304 | 3 rows Error rpc error code = ResourceExhausted desc = grpc received message larger than max 57552210 vs 4194304 make register Error 1``` 4194304 is similar to 4 1024 1024 and this is my mem limits for task set up in `flyte admin base config` But i m confused here then i saw a 57552210 ~ 54 Gb while trying to register a workflow proto file ` tmp register2016944902 2 app workflow main flow 2 pb` If you remember i can easily run my workflow locally and the max mem consumption is not more than 4Gb