hi `@channel`, the core Flyte team has heard repor...
# ask-the-community
n
hi
@channel
, the core Flyte team has heard reports of long import times when simply running a script locally that does
import flytekit
(the number I’ve heard of is ~8 seconds). I’ve been unable to reproduce this locally, but can anyone with this experience provide instructions on how to reproduce this behavior? We’d like to look into reducing this import time.
b
@Niels Bantilan I’ve done some digging, however I’ve done it on k8s. I’ve seen similarly long runtimes and to me it was mostly imports. Running a task with
PYTHONPROFILEIMPORTTIME=1
produced the attached logs. You should be able to produce an interactive flame-graph using
tuna
(docs).
Copy code
tuna flyte_profile.log
Seems like most of it is related to serialization and FlyteDecks - so potentially if you import FlyteDecks locally it could be slow? 🤔
n
thanks @Bernhard Stadlbauer this is awesome!
b
This was on a regular GC e2 machine with what I believe was
flytekit 1.4.x
(not 100% certain though)
k
cc @Yee this is amazing @Bernhard Stadlbauer thank you cc @Eduardo Apolinario (eapolinario) as well. we will be looking into it
@Bernhard Stadlbauer can you also share the code snippet for the same?
it seems flytedecks is enabled?
b
This was an empty task, no Flyte decks used (unless I’m missing a flag? 🤔 ) i.e. something along the following (untested):
Copy code
@task(
    environment={"PYTHONPROFILEIMPORTTIME": "1"}
)
def demo_task():
    print("foobar")

@workflow 
def demo_workflow():
    demo_task()
And then I’ve copied the import time logs from k8s
Happy to do some more debugging tomorrow
k
ohh
this is interesting
i see the bug, that flytedeck is being implicitly still invoked
we should solve that problem
cc @Kevin Su
128 Views