Mohd Shahid Khan Afridi
02/10/2023, 12:11 PMmessage: >-
failed at Node[n0]. RuntimeExecutionError: failed during plugin execution,
caused by: failed to execute handle for plugin [tensorflow]: found no
current condition. Conditions: []
Dan Rammer (hamersaw)
02/10/2023, 1:42 PMQueued
rather than failing the task the correct functionality here?Mohd Shahid Khan Afridi
02/10/2023, 1:46 PMWorkflow[flytesnacks:development:kftensorflow.tf_mnist.mnist_tensorflow_workflow] failed. RuntimeExecutionError: max number of system retry attempts [51/50] exhausted. Last known status message: failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [tensorflow]: found no current condition. Conditions: []
Byron Hsu
02/10/2023, 3:23 PMMohd Shahid Khan Afridi
02/11/2023, 8:01 AMWorkflow[flytesnacks:development:kftensorflow.tf_mnist.mnist_tensorflow_workflow] failed. RuntimeExecutionError: max number of system retry attempts [51/50] exhausted. Last known status message: failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [tensorflow]: found no current condition. Conditions: []
How can I debug this? How can I make sure if my plugin is properly enabled?Byron Hsu
02/11/2023, 3:45 PMkubectl describe <your tfjob>
)
2. Check tfjob’s pod logs (kubectl logs <your tfjob pod
)
3. Check propeller pod logs
Here is the several methods i will use to debug