Hi all wave I have a problem of sharing data between tasks I

Question

Hi all wave I have a problem of sharing data between tasks I found a similar issue here in discussions ```Workflow flyte anti fraud ml development app workflow main flow failed RuntimeExecutionError max number of system retry attempts 31 30 exhausted Last known status message failed at Node n0 RuntimeExecutionError failed during plugin execution caused by error file is too large 28775519 bytes max allowed 10485760 bytes``` I added `max output size bytes` params to the `flyte propeller config` and wait to apply all changes before re submitting a new task `kubectl edit configmap n flyte flyte propeller config` My propeller section of `flyte propeller config` looks like ``` core yaml | manager pod application flytepropeller pod template container name flytepropeller pod template name flytepropeller template propeller max output size bytes 52428800 downstream eval duration 30s enable admin launcher true leader election enabled true lease duration 15s lock config map name propeller leader namespace flyte renew deadline 10s retry period 2s limit namespace all max workflow retries 3 metadata prefix metadata propeller metrics prefix flyte prof port 10254``` Task configuration has been set up via `kubectl n flyte edit cm flyte admin base config` ``` storage yaml | storage type minio container my s3 bucket stow kind s3 config access key id minio auth type accesskey secret key miniostorage disable ssl true endpoint region us east 1 signedUrl stowConfigOverride endpoint enable multicontainer false limits maxDownloadMBs 50 task resource defaults yaml | task resources defaults cpu 1 memory 3000Mi storage 100Mi limits cpu 4 gpu 1 memory 3Gi storage 500Mi``` Also changing `maxDownloadMBs` didn t change the situation Changing cache `max size mbs` in `flyte propeller config` from 0 to some custom value also not working ``` cache yaml | cache max size mbs 100 target gc percent 70``` I ty to change different time with different params but the error was arising during each new executions I saw that none of `max output size bytes` or `max workflow retries` changed from 30 gt 3 are passed to the workflow execution ```RuntimeExecutionError max number of system retry attempts 31 30 exhausted error file is too large 28775519 bytes max allowed 10485760 bytes ``` Hereafter are my cli steps to create a new execution ``` kubectl n flyte edit cm flyte admin base config kubectl edit configmap n flyte flyte propeller config flytectl get task resource attribute p flyteexamples d development flytectl update project p flyte anti fraud ml d development storage cache max size mbs 100 flytectl get launchplan project flyte anti fraud ml domain development app workflow main flow latest execFile exec spec yaml flytectl create execution project flyte anti fraud ml domain development execFile exec spec yaml``` What additional steps I have to do to force flytectl to use my propeller changes and solve the problem of a max 10Mb size allowed for serialized uploads to flyte

Samhita Alla · Answer

< Anthony> can you try restarting your propeller after editing your config map

Samhita Alla · Answer

I m not sure if this solves the problem though But no harm in giving that a try

Anthony · Answer

Im using registration and task submission via flytectl Also changing `kubectl edit configmap n flyte flyte propeller config` I have to wait 5 min for healthy upstream in flyte UI console i though that it was restarting automaticaly after new changes in config are submitted How could I forcibly restart `flyte propeller` via `kubectl cli`

Anthony · Answer

Im already using 2 steps in my Makefile `serialize` gt `register` ``` PHONY serialize serialize echo $ CURDIR pyflyte c flyte config pkgs app package force in container source path root image $ FULL IMAGE NAME $ VERSION PHONY register register docker push serialize register serialize flytectl c $ FLYTECTL CONFIG register files project $ PROJECT domain development archive flyte package tgz force version $ VERSION ``` Also according to i saw that some flags are deprecated in favour of using congif file via `flyte admin` ``` sourceUploadPath string Deprecated Update flyte admin to avoid having to configure storage access from flytectl ``` My problem that I have one task that generates a dataclasses instances on the exit and another task should takes these classes as input params ``` workflow def main flow gt Forecast Main Flyte WorkFlow consisting of three tasks preproc and split train xgboost clf get predictions log= START START Raw Preprocessing and Splitting timestamp=None train cls target cls = preproc and split log= START START Initialize Boosting Params timestamp=None saved mpath = train xgboost clf feat cls=train cls target cls=target cls xgb params=xgb params cust metric=BoostingCustMetric ``` Where ```def preproc and split gt Tuple Fraud Raw PostProc Data Class Fraud Raw Target Data Class ``` These dataclasses are produced only after task execution so i cannot register them as some proto files coz they are Promising return args for my function 10mb error arises during sharing these dataclasses between tasks And i m wondering of how to apply changes in `flyte propeller config` to pass these serialisation limitations

Samhita Alla · Answer

`kubectl n flyte rollout restart deploy flytepropeller` is the restart command

Anthony · Answer

lemme check this

Anthony · Answer

< Samhita Alla> by restarting flytepropeller i can solve my issue `kubectl n flyte rollout restart deploy flytepropeller`

Anthony · Answer

tada

Anthony · Answer

thank you