Hi all, just evaluating flyte right now and looks ...
# announcements
Hi all, just evaluating flyte right now and looks promising for automating and scaling our container based workflows! I'm wondering if there are plans to add support for local execution of raw containers?
๐Ÿ™ 2
hi @Felix Ruess firstly welcome. Thank you for joining
Raw container execution can be done locally, we have not had an issue or anyone asking, but it should not be too much work
please file an issue - you can even contribute ๐Ÿ™‚
it is purely in python and the execute function today just executes the script
We already support local execution now, but it will always return
. Do you want to start a docker container locally to run that raw_container task? Check https://github.com/flyteorg/flytekit/blob/386626356ff2cb810733a29a51cf3d6351c8424d/flytekit/core/container_task.py#L72-L81
Thanks for the quick reply! Yes, exactly since we already have our tasks containerized and they read/write all their data to minio, I would like to just run them locally when building a new workflow or I don't have production cluster set up yet.
Seems like it should be easy to add very basic local execution via subprocess then (assuming docker, etc).
yes, would you like to help us to update the
The demo/sandbox doesn't fit my basic first steps well, as I "just" need to run the containers (images are large and already locally available, but not in sandbox).
sure, I'll see what basic execution I can come up with.
Thank you, as you said we need to use subprocess to launch a docker container to run a task, and try to fetch the task output from that container and return it.
stdout and stderr I guess?
yes, I think so
ok, any pointers on where to check how success/error should be reported?
sorry, I think youโ€™re able to use Popen.communicate to get output and error. If we got an error, just raise an exception in flytekit
Copy code
bashCommand = f"docker run ...."
process = subprocess.Popen(bashCommand.split(), stdout=subprocess.PIPE)
outs, errs = proc.communicate()
ok, I'll poke around a bit first to get a bit familiar and then start with just raising an exception
๐Ÿ‘ 1