Hi I have a flyte task that runs a subprocess that...
# ask-the-communitya
Alykhan Tejani
12/12/2023, 2:58 PMHi I have a flyte task that runs a subprocess that runs a binary. I want to redirect the output of the subprocess s.t. it displays in flyte logs. Is this possible?
k
Ketan (kumare3)
12/12/2023, 3:14 PM
Use shell task - all of this is handled for you
a
Alykhan Tejani
12/12/2023, 3:15 PMI am using a TFJob in the task config
Alykhan Tejani
12/12/2023, 3:15 PMwill this still work with shell task?
Alykhan Tejani
12/12/2023, 3:15 PMor should I call the shellTask from within the other task
Alykhan Tejani
12/12/2023, 3:15 PMi.e. my task looks like:
Alykhan Tejani
12/12/2023, 3:16 PM Copy code
@task(
task_config=TfJob(
chief=Chief(replicas=1),
ps=PS(replicas=1),
worker=Worker(replicas=1),
),
)
def some_task_name(...):
<download binary>
subprocess.run(binary)k
Ketan (kumare3)
12/12/2023, 3:39 PM
No , in that case just redirect stdout and stderr
a
Alykhan Tejani
12/12/2023, 3:39 PMso just by doing
subprocess.run(binary, stdout=subprocess.PIPE)?Alykhan Tejani
12/12/2023, 3:40 PMif
binaryy
Yi Chiu
12/12/2023, 6:09 PMI think you can get the stdout and stderr with
Copy code
process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = process.communicate()k
Kevin Su
12/12/2023, 7:10 PMifyes, it will.is a long running binary (such as a training service) will this pipe logsbinary
2 Views