#3358 [Core feature] Ability to Capture Pipe stderr and stdout Issue created by rkrishnasanka Motivation: Why do you think this is important? Currently, I'm running algorithms that take over one week to run on the flight cluster; one of the best ways to keep track of the current status of the solver is to check the

stdout

. So the ability to pipe the

stdout

or the

stderr

to a

host:port

would be the best way to go about it. Goal: What should the final outcome look like, ideally? Being able to use a syntax like:

def stdout_pipe_callback(data: str):
   # send data to "<https://myfavoritehost:1234>"
   ...

def stderr_pipe_callback(data: str):
   # send data to "<https://myotherfavoritehost:1234>"
   ...

@stream_task(stdout_pipe=stdout_pipe_callback, stderr_pipe=stderr_pipe_callback)
def do_some_work(input:InputType) -> OutputType:
 .....
 return results

This would allow the user, to define custom callback methods to stream back the stdout. Describe alternatives you've considered I've considered using the shell task, but that won't allow me to "stream" the outputs until the end of the process. But I need help defining python code to use. Another alternative I'm considering is to figure out a way to modify the docker container to monitor all the python stdout. I haven't had any luck with that. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte