Question, does the logging from Flyte task only become available at a terminal state? Are there ways...

broad-train-34581

10/18/2023, 9:58 AM

Question, does the logging from Flyte task only become available at a terminal state? Are there ways to get the live logs beside writing the logs to a provider?

broad-train-34581

10/18/2023, 10:02 AM

im running the below and running

kubectl logs pod_name

on the cluster and expecting to see the logs on runtime but im not till the task ends

Copy code

@task
def print_task():
    print("Starting")
    duration = 5 * 60 
    interval = 30
    iterations = duration // interval

    for _ in range(iterations):
        print("Print")
        time.sleep(interval)

agreeable-kitchen-44189

10/18/2023, 12:52 PM

Hey Lee! This seems unrelated to Flyte, but could be related to when Python flushes lines. Would you mind changing your print statement to:

Copy code

print("Print", flush=True)

and set

PYTHONUNBUFFERED=1

as an environment variable to check whether that helps? You should be able to set the environment variable using the

environment

keyword in the

task()

decorator

🫡 1

thx 1

freezing-airport-6809

10/18/2023, 2:35 PM

Or in pyflyte run

broad-train-34581

10/19/2023, 6:05 AM

thanks! i tried using

flush=true

works, setting

PYTHONUNBUFFERED

wasn't required. Was trying to figure out why were all the ShellTask logging unresponsive and its probably due to this print without the flush too.

broad-train-34581

10/19/2023, 6:12 AM

In anycase i took a read and tried setting

PYTHONUNBUFFERED=1

for the ShellTask and they print as expected now. Thanks alot!

21 Views

Open in Slack

Previous Next

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.