https://flyte.org logo
#ask-the-community
Title
# ask-the-community
n

Niels Bantilan

04/11/2023, 5:59 PM
hi
@channel
, the core Flyte team has heard reports of long import times when simply running a script locally that does
import flytekit
(the number I’ve heard of is ~8 seconds). I’ve been unable to reproduce this locally, but can anyone with this experience provide instructions on how to reproduce this behavior? We’d like to look into reducing this import time.
b

Bernhard Stadlbauer

04/11/2023, 6:56 PM
@Niels Bantilan I’ve done some digging, however I’ve done it on k8s. I’ve seen similarly long runtimes and to me it was mostly imports. Running a task with
PYTHONPROFILEIMPORTTIME=1
produced the attached logs. You should be able to produce an interactive flame-graph using
tuna
(docs).
Copy code
tuna flyte_profile.log
Seems like most of it is related to serialization and FlyteDecks - so potentially if you import FlyteDecks locally it could be slow? 🤔
n

Niels Bantilan

04/11/2023, 6:58 PM
thanks @Bernhard Stadlbauer this is awesome!
b

Bernhard Stadlbauer

04/11/2023, 7:00 PM
This was on a regular GC e2 machine with what I believe was
flytekit 1.4.x
(not 100% certain though)
k

Ketan (kumare3)

04/11/2023, 7:29 PM
cc @Yee this is amazing @Bernhard Stadlbauer thank you cc @Eduardo Apolinario (eapolinario) as well. we will be looking into it
@Bernhard Stadlbauer can you also share the code snippet for the same?
it seems flytedecks is enabled?
b

Bernhard Stadlbauer

04/11/2023, 8:41 PM
This was an empty task, no Flyte decks used (unless I’m missing a flag? 🤔 ) i.e. something along the following (untested):
Copy code
@task(
    environment={"PYTHONPROFILEIMPORTTIME": "1"}
)
def demo_task():
    print("foobar")

@workflow 
def demo_workflow():
    demo_task()
And then I’ve copied the import time logs from k8s
Happy to do some more debugging tomorrow
k

Ketan (kumare3)

04/11/2023, 9:35 PM
ohh
this is interesting
i see the bug, that flytedeck is being implicitly still invoked
we should solve that problem
cc @Kevin Su
2 Views