https://flyte.org logo
#announcements
Title
# announcements
f

Felix Ruess

06/30/2022, 5:06 PM
Hi all, just evaluating flyte right now and looks promising for automating and scaling our container based workflows! I'm wondering if there are plans to add support for local execution of raw containers?
๐Ÿ™ 2
k

Ketan (kumare3)

06/30/2022, 5:27 PM
hi @Felix Ruess firstly welcome. Thank you for joining
Raw container execution can be done locally, we have not had an issue or anyone asking, but it should not be too much work
please file an issue - you can even contribute ๐Ÿ™‚
it is purely in python and the execute function today just executes the script
k

Kevin Su

06/30/2022, 5:29 PM
We already support local execution now, but it will always return
None
. Do you want to start a docker container locally to run that raw_container task? Check https://github.com/flyteorg/flytekit/blob/386626356ff2cb810733a29a51cf3d6351c8424d/flytekit/core/container_task.py#L72-L81
f

Felix Ruess

06/30/2022, 5:36 PM
Thanks for the quick reply! Yes, exactly since we already have our tasks containerized and they read/write all their data to minio, I would like to just run them locally when building a new workflow or I don't have production cluster set up yet.
Seems like it should be easy to add very basic local execution via subprocess then (assuming docker, etc).
k

Kevin Su

06/30/2022, 5:37 PM
yes, would you like to help us to update the
execution
function?
f

Felix Ruess

06/30/2022, 5:38 PM
The demo/sandbox doesn't fit my basic first steps well, as I "just" need to run the containers (images are large and already locally available, but not in sandbox).
sure, I'll see what basic execution I can come up with.
k

Kevin Su

06/30/2022, 5:43 PM
Thank you, as you said we need to use subprocess to launch a docker container to run a task, and try to fetch the task output from that container and return it.
f

Felix Ruess

06/30/2022, 5:45 PM
stdout and stderr I guess?
k

Kevin Su

06/30/2022, 5:46 PM
yes, I think so
f

Felix Ruess

06/30/2022, 5:57 PM
ok, any pointers on where to check how success/error should be reported?
k

Kevin Su

06/30/2022, 6:38 PM
sorry, I think youโ€™re able to use Popen.communicate to get output and error. If we got an error, just raise an exception in flytekit
Copy code
bashCommand = f"docker run ...."
process = subprocess.Popen(bashCommand.split(), stdout=subprocess.PIPE)
outs, errs = proc.communicate()
f

Felix Ruess

06/30/2022, 6:45 PM
ok, I'll poke around a bit first to get a bit familiar and then start with just raising an exception
๐Ÿ‘ 1
27 Views