Hi all…
I have a few process now converted into ta...
# ask-the-communitye
Eduardo Matus
04/18/2023, 7:11 PMHi all…
I have a few process now converted into tasks (just added the decorator) but when I run the task takes a lot of time to complete compared when I run the process in a non-flyte environment. I set the same resources.
Any ideas?
k
Ketan (kumare3)
04/18/2023, 7:33 PM
Hmm quantify lot of time
e
Eduardo Matus
04/18/2023, 8:34 PMnonflyte takes 50 mins aprox. on Flyte environment takes 2 hours aprox
k
Kevin Su
04/18/2023, 8:52 PMwhat kind of task you are running? model training? are you reading large files from s3?
e
Eduardo Matus
04/18/2023, 9:05 PM@Kevin Su. Read large data sets from elastic search to then transform the data and store in another index. The existing process has many multiprocessing steps.
k
Kevin Su
04/18/2023, 9:08 PMwhich version of flytekit you are running?
e
Eduardo Matus
04/18/2023, 9:09 PMversion = “1.2.0-b1”
k
Kevin Su
04/18/2023, 9:10 PMwe have a pr to add runtime metric in flytekit, so it will be easy to debug by using that. https://github.com/flyteorg/flytekit/pull/1581
Kevin Su
04/18/2023, 9:11 PMjust to confirm, you are using elastic search sdk to download data, not using awscli, right
e
Eduardo Matus
04/18/2023, 9:14 PMcorrect
k
Ketan (kumare3)
04/19/2023, 1:33 AM
@Eduardo Matus also I think you should add enough cpus
Ketan (kumare3)
04/19/2023, 1:33 AM
And memory
Ketan (kumare3)
04/19/2023, 1:34 AM
Hmm, can you add a couple log lines, I would be curious and want to help
Ketan (kumare3)
04/19/2023, 1:34 AM
If at all it should go faster with more power
f
Felix Ruess
04/19/2023, 8:26 AM@Eduardo Matus what do you mean by a "non-flyte environment"?
Still in containers on kubernetes though? And there on the same class of node/machine?
151 Views