Hi everyone! It's me again. So, I managed to solve the issues I was having when trying to make Flyte...

ambitious-france-31318

08/07/2023, 4:45 PM

Hi everyone! It's me again. So, I managed to solve the issues I was having when trying to make Flyte detect the MPI plugin. Right now I've managed to make a workflow start executing but it never finishes. The info showed on the Flytepropeller console shows the following error:

E0807 16:38:21.462968       1 workers.go:102] error syncing 'flytesnacks-development/fa6f1f44573a6451e9cb': failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [mpi]: found no current condition. Conditions: []

. If anybody knows what's that error is really telling me and what would be a nice idea to solve it I would really appreciate! 😄

glamorous-carpet-83516

08/07/2023, 5:15 PM

which version of MPI operator are you using

glamorous-carpet-83516

08/07/2023, 5:15 PM

did you use https://github.com/kubeflow/training-operator

ambitious-france-31318

08/07/2023, 6:18 PM

Yeap, im using that MPI operator

ambitious-france-31318

08/07/2023, 6:19 PM

I already tried with versions 1.4.0, 1.5.0 y 1.6.0

ambitious-france-31318

08/07/2023, 6:21 PM

Now I was going to try the 1.3.0 version, I read somewhere that that error msg might be due to some modifications from one version to another, but im not sure if that's the way to go 😅

ambitious-france-31318

08/07/2023, 6:24 PM

Just in case, I believe it's worth mentioning that I initially tried with the mpi-operator instead of the training-operator, but since I couldn't make it work (the workflow always stayed in

UNKNOWN

status) I started googling and read somewhere that I should use the training-operator instead

tall-lock-23197

08/08/2023, 5:25 AM

Yes, you need to use the training operator. I tried it out recently and it worked for me. Please follow the instructions outlined in this guide: https://docs.flyte.org/en/latest/deployment/plugins/k8s/index.html.

ambitious-france-31318

08/08/2023, 3:18 PM

I hope you don't mind but if it's okay with you, we can continue on this thread, im getting a bit dizzy jumping from one channel to another haha, perhaps it wasn't my brightest idea to ask the same thing on two different places 😅 Btw, I've been following the instructions on that guide with no luck

24 Views

Open in Slack

Previous Next

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.