Hi all, just evaluating flyte right now and looks promising for automating and scaling our container based workflows! I'm wondering if there are plans to add support for local execution of raw containers?

hi @Felix Ruess firstly welcome. Thank you for joining

We already support local execution now, but it will always return

None

. Do you want to start a docker container locally to run that raw_container task? Check https://github.com/flyteorg/flytekit/blob/386626356ff2cb810733a29a51cf3d6351c8424d/flytekit/core/container_task.py#L72-L81

Thanks for the quick reply! Yes, exactly since we already have our tasks containerized and they read/write all their data to minio, I would like to just run them locally when building a new workflow or I don't have production cluster set up yet.

yes, would you like to help us to update the

execution

function?

The demo/sandbox doesn't fit my basic first steps well, as I "just" need to run the containers (images are large and already locally available, but not in sandbox).

Thank you, as you said we need to use subprocess to launch a docker container to run a task, and try to fetch the task output from that container and return it.

stdout and stderr I guess?

yes, I think so

ok, any pointers on where to check how success/error should be reported?

sorry, I think you’re able to use Popen.communicate to get output and error. If we got an error, just raise an exception in flytekit

bashCommand = f"docker run ...."
process = subprocess.Popen(bashCommand.split(), stdout=subprocess.PIPE)
outs, errs = proc.communicate()

ok, I'll poke around a bit first to get a bit familiar and then start with just raising an exception