Hello, I'm running a simple workflow/task that loop during 5min and writes a log every 5 seconds. when I run that task locally it shows the logs in realtime. However when I run it remotely on my k8s cluster, the stdout of the pod is only visible at the very end of the execution (after 5minutes).

@task(requests=Resources(cpu="1", mem="500Mi"), limits=Resources(cpu="2", mem="1Gi"))
def say_hello() -> str:
    for i in range(60):
        print("writing some log", i)
        time.sleep(5)
    return "Hello, World!"

I run the task with:

pyflyte run --remote ./long_task.py long_wf

Probably because logs are buffeted and only flushed either when buffer is full or process exits. You can set buffer to zero, but if you are looking at some logging service they may aggregate at fluent bit etc. logging in super realtime. Is too expensive

so we need to have real time logs visible to the end user (in the UI). how do you configure the buffer to 0 ? I didn't find any doc about that. if flytekit process holds the underlying task logs, we won't be able to collect it on time at the pod level (using fluentbit or whatever other system)

thanks, but that's for python processes. when I run my task locally I don't get any buffering issue. is there a way to configure it on flyte ?

Env var

thanks, that worked!