Fabio Grätz
06/06/2023, 3:46 PMBernhard Stadlbauer
06/06/2023, 3:55 PMdask
in case folks want to have a different image in one of the components. Docs (and what takes precedence) hereFabio Grätz
06/06/2023, 4:01 PMByron Hsu
06/06/2023, 4:02 PMYubo Wang
06/06/2023, 4:03 PMFabio Grätz
06/06/2023, 4:04 PMpyflyte run
I guess?Yubo Wang
06/06/2023, 4:04 PMFabio Grätz
06/06/2023, 4:07 PMfrom flytekit import task, workflow, Resources
from flytekitplugins.kfpytorch import Elastic, PyTorch
@task(
task_config=PyTorch(
num_workers=1
),
requests=Resources(cpu="3", mem="3Gi"),
)
def train():
print("Trained")
@workflow
def wf():
train()
When I run this workflow with flytekit==1.6.2
installed locally it runs, propeller image is also version 1.6.2.
When I install latest master of flytekit, the task stays queued with this error:
E0606 16:05:16.771720 1 workers.go:102] error syncing 'development/f0024b9843ea14ed3b8c': failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [pytorch]: number of worker should be more then 0
I guess to use the new pytorch task config I’ll need a new propeller image, right?
Do we already have an image built in CI somewhere?Yubo Wang
06/06/2023, 4:09 PMFabio Grätz
06/06/2023, 4:11 PMYubo Wang
06/06/2023, 4:11 PMFabio Grätz
06/06/2023, 4:11 PMYubo Wang
06/06/2023, 4:15 PMFabio Grätz
06/06/2023, 4:21 PM❯ k -n flyte get pod flytepropeller-668549d8bd-srqk2 -o yaml | grep image
image: <http://cr.flyte.org/flyteorg/flytepropeller-release:v1.6.2|cr.flyte.org/flyteorg/flytepropeller-release:v1.6.2>
flyte
repo not in propeller.Yubo Wang
06/06/2023, 4:24 PMFabio Grätz
06/06/2023, 4:25 PMYubo Wang
06/06/2023, 4:25 PMFabio Grätz
06/06/2023, 4:26 PMI think the 1.6.2 should already include the changes I madeIf this is the case (didn’t check), then something is broken, right?
Yubo Wang
06/06/2023, 4:28 PMFabio Grätz
06/06/2023, 4:30 PMEduardo Apolinario (eapolinario)
06/06/2023, 6:01 PMYubo Wang
06/07/2023, 7:17 AMFabio Grätz
06/07/2023, 8:02 AMYubo Wang
06/07/2023, 8:42 AMFabio Grätz
06/07/2023, 9:11 AMYubo Wang
06/08/2023, 2:29 AMFabio Grätz
06/08/2023, 7:54 AMYubo Wang
06/08/2023, 7:56 AMFabio Grätz
06/08/2023, 7:11 PM