hi

@channel

, the core Flyte team has heard reports of long import times when simply running a script locally that does

import flytekit

(the number I’ve heard of is ~8 seconds). I’ve been unable to reproduce this locally, but can anyone with this experience provide instructions on how to reproduce this behavior? We’d like to look into reducing this import time.

@Niels Bantilan I’ve done some digging, however I’ve done it on k8s. I’ve seen similarly long runtimes and to me it was mostly imports. Running a task with

PYTHONPROFILEIMPORTTIME=1

produced the attached logs. You should be able to produce an interactive flame-graph using

tuna

(docs).

tuna flyte_profile.log

Seems like most of it is related to serialization and FlyteDecks - so potentially if you import FlyteDecks locally it could be slow? 🤔

thanks @Bernhard Stadlbauer this is awesome!

This was on a regular GC e2 machine with what I believe was

flytekit 1.4.x

(not 100% certain though)

cc @Yee this is amazing @Bernhard Stadlbauer thank you cc @Eduardo Apolinario (eapolinario) as well. we will be looking into it

This was an empty task, no Flyte decks used (unless I’m missing a flag? 🤔 ) i.e. something along the following (untested):

@task(
    environment={"PYTHONPROFILEIMPORTTIME": "1"}
)
def demo_task():
    print("foobar")

@workflow 
def demo_workflow():
    demo_task()

And then I’ve copied the import time logs from k8s

ohh