https://flyte.org
Join Slack
[RESOLVED] Facing below error while trying to use podTemplate for a TfJob task. ```Workflow[moj-ml-w...
# flyte-support
g
[RESOLVED] Facing below error while trying to use podTemplate for a TfJob task.
Copy code
Workflow[moj-ml-workflows-flyte-example:development:flyte.example.tfjob.workflow.mnist_tensorflow_workflow] failed. RuntimeExecutionError: max number of system retry attempts [51/50] exhausted. Last known status message: failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [tensorflow]: [BadTaskSpecification] Unable to create pod spec: [[BadTaskSpecification] PodTemplate 'flyte-pod-template' does not exist]
Have created podTemplate using below yaml file:
Copy code
apiVersion: v1
kind: PodTemplate
metadata:
  name: flyte-pod-template
  namespace: development
template:
  metadata:
    labels:
      foo: testLabel
    annotations:
      foo: initial-value
      bar: initial-value
  spec:
    containers:
      - name: default
        image: <http://docker.io/rwgrim/docker-noop|docker.io/rwgrim/docker-noop>
        terminationMessagePath: "/dev/foo"
Verified it in corresponding k8s namespace:
Copy code
kubectl get podTemplate -n development

NAME         CONTAINERS  IMAGES             POD LABELS
flyte-pod-template  default   <http://docker.io/rwgrim/docker-noop|docker.io/rwgrim/docker-noop>  foo=testLabel
Copy code
kubectl get podTemplate -n flyte

NAME         CONTAINERS  IMAGES             POD LABELS
flyte-pod-template  default   <http://docker.io/rwgrim/docker-noop|docker.io/rwgrim/docker-noop>  foo=templateLabel
Task definintion:
t
I think the pod template needs to be present in the namespace where the execution is triggered.
g
The pods are getting created in 
development
namespace and flyte cluster is setup in 
flyte
namespace both has the podTemplate present with this name
t
It needs to be present in the flytesnacks-development namespace (assuming flytesnacks is the project you're triggering your execution in)
g
For me the executions/flyte-pods are getting created in just 
development
namespace, no project name prefixed
Copy code
kubectl get <http://flyteworkflows.flyte.lyft.com|flyteworkflows.flyte.lyft.com> -n development
NAME          AGE
af80ebe30d4426b22c38  10h
ar22wvbzk49qrc7b2htn  3m24s
ffdf33fba0b3b9f42000  6h49m
UI view for my example project, all the successful/failed executions go into 
development
namespace
g
I ran into the same issue recently, this is due to a bug. I am not an my PC just now but I will link the thread when I am. The fix however is to create a completely default pod template and set it as the default pod template in the flyte config, then Flyte will start picking up all the templates and you should be good
g
Aah, the default template doesn't serve my purpose though 😞 ... I needed custom labels for the individual tasks. For TfJobs the labels aren't getting propagated from launchplan or so, the task level podTemplate was my only hope
Do you mean one default template in config will help the cluster to pick these individual task template as well?
👍 1
g
You misunderstand.... You need to set a default flyte wide podTemplate, and then you can use a task level template. It's an issue where Flyte doesn't start looking for templates at all unless a default is set
g
Got it...on to it 🤞
g
Give it a go and you will see what I mean 😀
🏃‍♂️ 1
c
There is a hierarchy. You can have a pod template in the flyte namespace, then one in the namespace of the execution. Then on the task.
g
Thanks @gray-ocean-62145, it worked!!
🙌 1
Thanks @tall-lock-23197 @cool-lifeguard-49380
👍 1
21 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by