Hi all… I have a few process now converted into tasks (just added the decorator) but when I run the task takes a lot of time to complete compared when I run the process in a non-flyte environment. I set the same resources. Any ideas?

Hmm quantify lot of time

nonflyte takes 50 mins aprox. on Flyte environment takes 2 hours aprox

what kind of task you are running? model training? are you reading large files from s3?

@glamorous-carpet-83516. Read large data sets from elastic search to then transform the data and store in another index. The existing process has many multiprocessing steps.

which version of flytekit you are running?

version = “1.2.0-b1”

we have a pr to add runtime metric in flytekit, so it will be easy to debug by using that. https://github.com/flyteorg/flytekit/pull/1581

correct

@rhythmic-lizard-91902 also I think you should add enough cpus

@rhythmic-lizard-91902 what do you mean by a "non-flyte environment"? Still in containers on kubernetes though? And there on the same class of node/machine?