delightful-computer-49028
11/18/2022, 11:43 PMglamorous-carpet-83516
11/19/2022, 12:05 AMglamorous-carpet-83516
11/19/2022, 12:05 AMconfigmap:
enabled_plugins:
# -- Task specific configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#GetConfig)
tasks:
# -- Plugins configuration, [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#TaskPluginConfig)
task-plugins:
# -- [Enabled Plugins](https://pkg.go.dev/github.com/flyteorg/flyteplugins/go/tasks/config#Config). Enable SageMaker*, Athena if you install the backend
# plugins
enabled-plugins:
- container
- sidecar
- k8s-array
- mpi
default-for-task-types:
container: container
sidecar: sidecar
container_array: k8s-array
mpi: mpiglamorous-carpet-83516
11/19/2022, 12:05 AMdelightful-computer-49028
11/19/2022, 12:05 AMfreezing-airport-6809
freezing-airport-6809
tall-lock-23197
great-school-54368
11/21/2022, 4:44 AMdelightful-computer-49028
11/21/2022, 5:35 PMhttps://github.com/flyteorg/flytesnacks/releases/download/v0.3.112/snacks-cookbook-integrations-kubernetes-kfmpi.tar.gzgreat-school-54368
11/23/2022, 2:15 PMfreezing-airport-6809
freezing-airport-6809
freezing-airport-6809
delightful-computer-49028
11/23/2022, 3:51 PMkubectl -n flyte get crd
NAME CREATED AT
backendconfigs.cloud.google.com 2022-11-15T00:00:37Z
capacityrequests.internal.autoscaling.gke.io 2022-11-15T00:00:07Z
flyteworkflows.flyte.lyft.com 2022-11-15T02:05:53Z
frontendconfigs.networking.gke.io 2022-11-15T00:00:38Z
managedcertificates.networking.gke.io 2022-11-15T00:00:24Z
memberships.hub.gke.io 2022-11-15T00:03:55Z
mpijobs.kubeflow.org 2022-11-18T17:42:49Z
serviceattachments.networking.gke.io 2022-11-15T00:00:41Z
servicenetworkendpointgroups.networking.gke.io 2022-11-15T00:00:39Z
updateinfos.nodemanagement.gke.io 2022-11-15T00:00:40Z
volumesnapshotclasses.snapshot.storage.k8s.io 2022-11-15T00:00:38Z
volumesnapshotcontents.snapshot.storage.k8s.io 2022-11-15T00:00:39Z
volumesnapshots.snapshot.storage.k8s.io 2022-11-15T00:00:39Zfreezing-airport-6809
freezing-airport-6809
delightful-computer-49028
11/23/2022, 3:55 PMgreat-school-54368
11/24/2022, 1:52 PM{
"json": {
"exec_id": "aggfj525qt8stl9lnqk9",
"ns": "flytesnacks-development",
"res_ver": "4206",
"routine": "worker-1",
"wf": "flytesnacks:development:kfmpi.mpi_mnist.horovod_training_wf"
},
"level": "error",
"msg": "Error when trying to reconcile workflow. Error [failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [mpi]: found no current condition. Conditions: []]. Error Type[*errors.NodeErrorWithCause]",
"ts": "2022-11-24T13:48:59Z"
}
Propeller Config
enabled_plugins.yaml: |
tasks:
task-plugins:
default-for-task-types:
container: container
container_array: k8s-array
mpi: mpi
sidecar: sidecar
enabled-plugins:
- container
- sidecar
- k8s-array
- mpi
@hallowed-mouse-14616 Any idea what can be the issue ?freezing-airport-6809
great-school-54368
11/25/2022, 2:21 PMhallowed-mouse-14616
12/01/2022, 3:50 PM