Hello after deploying default k8 pod templates via helm the namespaces do not seem to be created. I can create new projects but any attempts to deploy a workflow give "failed to create workflow in propeller namespaces [] not found". There do not seem to be any errors in the logs

do you have the resource controller working?

@boundless-lifeguard-61788 what Helm chart are you using? The PodTemplate and namespace creation are two separate operations, configurable on different sections in the chart

this is a custom setup. The issue might not be related to the pod template. 2 changes happened before this issue: I deployed a default pod template and at the same time the postgresql db had to be restored from backup, It was working fine before then. I can create a project but no namespace is created and there are not errors before attempting to deploy a workflow.

clusterResourceTemplates = {
      inline = {
        "my-template.yaml" = file("path to template")
      }
    }
....        
plugins = {
          k8s = {
            default-pod-template-name = "my-template"
           
          }
        }

can you verify if the

sync-cluster-resources

Pod is Running?

under what namespace should I expect to see this pod?

the same where Flyte is runnning, typically

flyte

no its not there, even after reverting and the namespaces being generated correctly

this is true if you're running the

flyte-core

chart though

After removing the default pod template the namespaces the issue is resolved, I believe this might be a misconfiguration of the default template. I was following the docs here https://docs.flyte.org/en/latest/deployment/configuration/general.html#runtime-podtemplates

Ah right. Could you file an Issue please? I see the mermaid code there but the diagram is not rendering

I have a single flyte-binary deployment. As a test if I add a basic skeleton "noop" pod template then the namespace fails to create. "If no default configuration is desired, specifying a container with a name other than “default” or “primary” (for example, “noop”) is considered best practice. Since Flyte only processes the “default” or “primary” containers, this value will always be dropped during Pod construction." template below and the terraform is the same above

apiVersion: v1
kind: PodTemplate
metadata:
  name: test
  namespace: flyte
template:
  metadata:
    labels:
      foo: from-pod-template
  spec:
    containers:
      - name: noob # first test, this should be ignored
        image: <http://docker.io/rwgrim/docker-noop|docker.io/rwgrim/docker-noop>

I've spent like all day digging this and I think it's time to file an Issue

where do you file an issue

[flyte-bug]

🐞 Create a new Flyte Bug issue: https://github.com/flyteorg/flyte/issues/new?assignees=&labels=bug%2Cuntriaged&template=bug_report.yaml&title=%5BBUG%5D+

Thank you