https://flyte.org logo
#ask-the-community
Title
# ask-the-community
e

Eduardo Matus

04/18/2023, 7:11 PM
Hi all… I have a few process now converted into tasks (just added the decorator) but when I run the task takes a lot of time to complete compared when I run the process in a non-flyte environment. I set the same resources. Any ideas?
k

Ketan (kumare3)

04/18/2023, 7:33 PM
Hmm quantify lot of time
e

Eduardo Matus

04/18/2023, 8:34 PM
nonflyte takes 50 mins aprox. on Flyte environment takes 2 hours aprox
k

Kevin Su

04/18/2023, 8:52 PM
what kind of task you are running? model training? are you reading large files from s3?
e

Eduardo Matus

04/18/2023, 9:05 PM
@Kevin Su. Read large data sets from elastic search to then transform the data and store in another index. The existing process has many multiprocessing steps.
k

Kevin Su

04/18/2023, 9:08 PM
which version of flytekit you are running?
e

Eduardo Matus

04/18/2023, 9:09 PM
version = “1.2.0-b1”
k

Kevin Su

04/18/2023, 9:10 PM
we have a pr to add runtime metric in flytekit, so it will be easy to debug by using that. https://github.com/flyteorg/flytekit/pull/1581
just to confirm, you are using elastic search sdk to download data, not using awscli, right
e

Eduardo Matus

04/18/2023, 9:14 PM
correct
k

Ketan (kumare3)

04/19/2023, 1:33 AM
@Eduardo Matus also I think you should add enough cpus
And memory
Hmm, can you add a couple log lines, I would be curious and want to help
If at all it should go faster with more power
f

Felix Ruess

04/19/2023, 8:26 AM
@Eduardo Matus what do you mean by a "non-flyte environment"? Still in containers on kubernetes though? And there on the same class of node/machine?
2 Views