https://flyte.org
Join Slack
Hi all, just evaluating flyte right now and looks promising for automating and scaling our container...
# announcements
q
Hi all, just evaluating flyte right now and looks promising for automating and scaling our container based workflows! I'm wondering if there are plans to add support for local execution of raw containers?
🙏 2
f
hi @quaint-diamond-37493 firstly welcome. Thank you for joining
Raw container execution can be done locally, we have not had an issue or anyone asking, but it should not be too much work
please file an issue - you can even contribute 🙂
it is purely in python and the execute function today just executes the script
g
We already support local execution now, but it will always return 
None
. Do you want to start a docker container locally to run that raw_container task? Check https://github.com/flyteorg/flytekit/blob/386626356ff2cb810733a29a51cf3d6351c8424d/flytekit/core/container_task.py#L72-L81
q
Thanks for the quick reply! Yes, exactly since we already have our tasks containerized and they read/write all their data to minio, I would like to just run them locally when building a new workflow or I don't have production cluster set up yet.
Seems like it should be easy to add very basic local execution via subprocess then (assuming docker, etc).
g
yes, would you like to help us to update the 
execution
function?
q
The demo/sandbox doesn't fit my basic first steps well, as I "just" need to run the containers (images are large and already locally available, but not in sandbox).
sure, I'll see what basic execution I can come up with.
g
Thank you, as you said we need to use subprocess to launch a docker container to run a task, and try to fetch the task output from that container and return it.
q
stdout and stderr I guess?
g
yes, I think so
q
ok, any pointers on where to check how success/error should be reported?
g
sorry, I think you’re able to use Popen.communicate to get output and error. If we got an error, just raise an exception in flytekit
Copy code
bashCommand = f"docker run ...."
process = subprocess.Popen(bashCommand.split(), stdout=subprocess.PIPE)
outs, errs = proc.communicate()
q
ok, I'll poke around a bit first to get a bit familiar and then start with just raising an exception
👍 1
160 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by