Michael Tinsley08/09/2023, 12:53 PM
namespace - this is where the Flyte binary is deployed. I’ve also deployed this in the Flyte project/domain workspace, but the same issue happens.
Then I try and use it with a super simple test:
apiVersion: v1 kind: PodTemplate metadata: name: flyte-template-test namespace: flyte-backend template: spec: containers: - name: default image: <http://docker.io/rwgrim/docker-noop|docker.io/rwgrim/docker-noop> terminationMessagePath: "/dev/foo" hostNetwork: false
Submit it, and it errors with
@task(pod_template_name="flyte-template-test") def check() -> bool: return True
I’ve been through the permissions, and the default cluster role should have permission?
Workflow[playground:development:using_templates.template_wf.gpu_workflow] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: [BadTaskSpecification] PodTemplate 'flyte-template-test' does not exist
Does anyone have any pointers? Should be able to do some really cool stuff once I’ve got this working
... resources: - podtemplates - verbs: - create - delete - deletecollection - get - list - patch - post - update - watch
Gopal Vashishtha08/09/2023, 12:58 PM
Michael Tinsley08/09/2023, 1:00 PM
Gopal Vashishtha08/09/2023, 1:02 PM
Michael Tinsley08/09/2023, 1:20 PM
namespace, all pretty much default as per the flyte-binary helm chart 😬
Gopal Vashishtha08/09/2023, 1:27 PM
Michael Tinsley08/09/2023, 1:27 PM
Dan Rammer (hamersaw)08/09/2023, 3:48 PM
registered PodTemplate '%s:%s' in store
is the namespace and PodTemplate name if it is added to the cache. If this doesn't exist, then something is blocking propeller from seeing it.
Michael Tinsley08/10/2023, 9:59 AM
Dan Rammer (hamersaw)08/10/2023, 2:24 PM
Is there any setting that needs enabling for PodTemplates?There shouldn't be. Propeller uses a k8s resource watch to capture PodTemplate creations, updates, and deletes. Maybe you could try to emulate a watch using kubectl (something like
) with the same permissions as propeller and see if the PodTemplate changes are viewable. It sounds like for some reason propeller isn't being notified of updates from the k8s apiserver.
kubectl get --watch ...
Gopal Vashishtha08/11/2023, 2:51 AM
configuration: inline: plugins: k8s: default-pod-template-name:
Michael Tinsley08/11/2023, 9:51 AM
Dan Rammer (hamersaw)08/11/2023, 12:32 PM
value was set. However, with the introduction of task-level PodTemplates this should be running continuously.
Michael Tinsley08/11/2023, 12:42 PM
Victor Delépine08/11/2023, 9:23 PM