https://flyte.org logo
#ask-the-community
Title
# ask-the-community
n

Nabil Koroghli

02/15/2024, 9:57 AM
Hello, I'm running a simple workflow/task that loop during 5min and writes a log every 5 seconds. when I run that task locally it shows the logs in realtime. However when I run it remotely on my k8s cluster, the stdout of the pod is only visible at the very end of the execution (after 5minutes).
Copy code
@task(requests=Resources(cpu="1", mem="500Mi"), limits=Resources(cpu="2", mem="1Gi"))
def say_hello() -> str:
    for i in range(60):
        print("writing some log", i)
        time.sleep(5)
    return "Hello, World!"
I run the task with:
pyflyte run --remote ./long_task.py long_wf
k

Ketan (kumare3)

02/15/2024, 3:17 PM
Probably because logs are buffeted and only flushed either when buffer is full or process exits. You can set buffer to zero, but if you are looking at some logging service they may aggregate at fluent bit etc. logging in super realtime. Is too expensive
n

Nabil Koroghli

02/15/2024, 3:22 PM
so we need to have real time logs visible to the end user (in the UI). how do you configure the buffer to 0 ? I didn't find any doc about that. if flytekit process holds the underlying task logs, we won't be able to collect it on time at the pod level (using fluentbit or whatever other system)
n

Nabil Koroghli

02/15/2024, 3:27 PM
thanks, but that's for python processes. when I run my task locally I don't get any buffering issue. is there a way to configure it on flyte ?
k

Ketan (kumare3)

02/15/2024, 4:09 PM
Env var
Pyflyte run —env
n

Nabil Koroghli

02/15/2024, 4:15 PM
thanks, that worked!
although it would be nice to configure that through flyte sdk (task metadata or some global config?)
2 Views