https://flyte.org logo
Join Slack
Powered by
I have flyte-binary installed in my kubernetes clu...
# flyte-support
f
I have flyte-binary installed in my kubernetes cluster and it works great. But when I execute the hello-world pyhton test example it takes like 1 minute to execute that. Is there a way for smaller tasks to make it faster?
a
Usually most of the overhead is on container start times Could you check on your Timeline view and hover over the task to look at where time is spent?
Optimizing execution time in Flyte is done at multiple layers, like described in the docs but as usual is important to understand first where the bottleneck is. This is, for example, the first execution of the hello world example on a fresh cluster: `TASK_SETUP`usually indicate K8s API server Pod creation calls, Pod scheduling time, container image pull duration and in general everything K8s (or the particular plugin you may be using) does to prepare the execution environment. This is solved in Union where using `@actor`you get a reusable container (zero cold startup time) plus a remote container Image builder.
f
Hi, it says 3 sec for execution but now where the rest of the time went.
Copy code
from flytekit import task, workflow
@task
def say_hello() -> str:
    return "hello world"
@workflow
def my_wf() -> str:
    res = say_hello()
    return res
if __name__ == "__main__":
    print(f"Running my_wf() {my_wf()}")
òr is there maybe a better example
Had to reload it.
What is 1m Task Runtime?
a
Task Runtime is the actual duration of running the user code
f
for a hello_world?
or is loading the libs take so long
What is the best example to use
a
it includes flytekit downloading inputs and writing outputs to blobstore
f
ok. so the image is cached, but the flytekit needs to be downloaded every time?
is there a way to cache libs?
or is there an image that has that already included
a
no no, when you register a task for execution, the compilation is uploaded to blob storage from where flyte downloads inputs and where it persist outputs. That communication introduces some latency
ImageSpec caches by default, only rebuilding if there's a change
f
is it the compilation phase that takes so long?
Your execution TASK_RUNTIME only takes 9s why is it much longer with me.
a
f
But why does it take only 9s for you? Do have a different image or why?
Or do you cache?
a
Copy code
import typing
from flytekit import task, workflow


@task
def say_hello(name: str) -> str:
    return f"hello {name}!"


@task
def greeting_length(greeting: str) -> int:
    return len(greeting)

@workflow
def wf(name: str = "union") -> typing.Tuple[str, int]:
    greeting = say_hello(name=name)
    greeting_len = greeting_length(greeting=greeting)
    return greeting, greeting_len


if __name__ == "__main__":
    # Execute the workflow, simply by invoking it like a function and passing in
    # the necessary parameters
    print(f"Running wf() { wf(name='passengers') }")
this is what I ran. It
It's a freshly installed cluster and no caching
flytekit 1.14.0b6
f
image.png
s
@fast-salesclerk-89694 can you try assigning some higher memory request to the task like below ? Ive observed this as well with flyte python tasks, when you don't assign requests the task usually takes more time to start execution 
requests=Resources(cpu="300m", mem="700Mi"),
f
Wow. Your are right. It reduced from 60second to 6sec! This should be changed in the documentation as for newcomer this looks bad.
from flytekit import Resources, task, workflow @task(requests=Resources(cpu="900m", mem="3Gi")) def say_hello() -> str: return "hello world" @workflow def my_wf() -> str: res = say_hello() return res if name == "__main__": print(f"Running my_wf() {my_wf()}")
s
yeah, seems like a bug as the default requests are mentioned in values/config.
35 Views
Open in Slack
PreviousNext
Powered by