Tarmily Wen
11/18/2022, 11:43 PMKevin Su
11/19/2022, 12:05 AMconfigmap:
enabled_plugins:
# -- Task specific configuration [structure](<https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#GetConfig>)
tasks:
# -- Plugins configuration, [structure](<https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#TaskPluginConfig>)
task-plugins:
# -- [Enabled Plugins](<https://pkg.go.dev/github.com/flyteorg/flyteplugins/go/tasks/config#Config>). Enable SageMaker*, Athena if you install the backend
# plugins
enabled-plugins:
- container
- sidecar
- k8s-array
- mpi
default-for-task-types:
container: container
sidecar: sidecar
container_array: k8s-array
mpi: mpi
Tarmily Wen
11/19/2022, 12:05 AMKetan (kumare3)
11/19/2022, 12:07 AMSamhita Alla
11/21/2022, 3:57 AMYuvraj
11/21/2022, 4:44 AMTarmily Wen
11/21/2022, 5:35 PM<https://github.com/flyteorg/flytesnacks/releases/download/v0.3.112/snacks-cookbook-integrations-kubernetes-kfmpi.tar.gz>
Yuvraj
11/23/2022, 2:15 PMKetan (kumare3)
11/23/2022, 3:38 PMTarmily Wen
11/23/2022, 3:51 PMkubectl -n flyte get crd
NAME CREATED AT
<http://backendconfigs.cloud.google.com|backendconfigs.cloud.google.com> 2022-11-15T00:00:37Z
<http://capacityrequests.internal.autoscaling.gke.io|capacityrequests.internal.autoscaling.gke.io> 2022-11-15T00:00:07Z
<http://flyteworkflows.flyte.lyft.com|flyteworkflows.flyte.lyft.com> 2022-11-15T02:05:53Z
<http://frontendconfigs.networking.gke.io|frontendconfigs.networking.gke.io> 2022-11-15T00:00:38Z
<http://managedcertificates.networking.gke.io|managedcertificates.networking.gke.io> 2022-11-15T00:00:24Z
<http://memberships.hub.gke.io|memberships.hub.gke.io> 2022-11-15T00:03:55Z
<http://mpijobs.kubeflow.org|mpijobs.kubeflow.org> 2022-11-18T17:42:49Z
<http://serviceattachments.networking.gke.io|serviceattachments.networking.gke.io> 2022-11-15T00:00:41Z
<http://servicenetworkendpointgroups.networking.gke.io|servicenetworkendpointgroups.networking.gke.io> 2022-11-15T00:00:39Z
<http://updateinfos.nodemanagement.gke.io|updateinfos.nodemanagement.gke.io> 2022-11-15T00:00:40Z
<http://volumesnapshotclasses.snapshot.storage.k8s.io|volumesnapshotclasses.snapshot.storage.k8s.io> 2022-11-15T00:00:38Z
<http://volumesnapshotcontents.snapshot.storage.k8s.io|volumesnapshotcontents.snapshot.storage.k8s.io> 2022-11-15T00:00:39Z
<http://volumesnapshots.snapshot.storage.k8s.io|volumesnapshots.snapshot.storage.k8s.io> 2022-11-15T00:00:39Z
Ketan (kumare3)
11/23/2022, 3:52 PMTarmily Wen
11/23/2022, 3:55 PMYuvraj
11/24/2022, 1:52 PM{
"json": {
"exec_id": "aggfj525qt8stl9lnqk9",
"ns": "flytesnacks-development",
"res_ver": "4206",
"routine": "worker-1",
"wf": "flytesnacks:development:kfmpi.mpi_mnist.horovod_training_wf"
},
"level": "error",
"msg": "Error when trying to reconcile workflow. Error [failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [mpi]: found no current condition. Conditions: []]. Error Type[*errors.NodeErrorWithCause]",
"ts": "2022-11-24T13:48:59Z"
}
Propeller Config
enabled_plugins.yaml: |
tasks:
task-plugins:
default-for-task-types:
container: container
container_array: k8s-array
mpi: mpi
sidecar: sidecar
enabled-plugins:
- container
- sidecar
- k8s-array
- mpi
@Dan Rammer (hamersaw) Any idea what can be the issue ?Ketan (kumare3)
11/24/2022, 3:32 PMYuvraj
11/25/2022, 2:21 PMDan Rammer (hamersaw)
12/01/2022, 3:50 PM