We are running Flyte on GKE cluster, we use podTem...
# ask-the-community
a
We are running Flyte on GKE cluster, we use podTemplate to configure env vars and secrets. After updading the podTemplate, we were trying to run workflows using
pyflyte --image
but we’re getting this error (occasionally) -
Failed to apply default image tag "{{.image.demo.fqn}}:{{.image.demo.version}}": couldn't parse image reference "{{.image.demo.fqn}}:{{.image.demo.version}}": invalid reference format
Does anyone have any idea why it happens? Tnx
j
Looks like it didn’t render the image uri at all. What was the full command you ran? Please redact as necessary.
n
Run was done using the gui, same tasks were registered using this command:
Copy code
pyflyte register -p project_name -d development --image demo=someimeage:latest workflows/wf.py
Issue happens after updating podTemplate. same wfs run normally before.
j
Hmm. Can you paste the pod template as well? What do you mean when you say the error happens occasionally?
a
Copy code
apiVersion: v1
kind: PodTemplate
metadata:
  name: default
  namespace: dataengineer-development
template:
  metadata:
  spec:
    containers:
      - name: primary
        image: <http://docker.io/rwgrim/docker-noop|docker.io/rwgrim/docker-noop> # This value will always be overridden by Flyte
        env:
        - name: DB_USER
          valueFrom:
            secretKeyRef:
              name: db-non-prod
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-non-prod
              key: password
This is the pod template. It sometime works, sometimes not. We noticed happens after changing the pod tempalte
k
Why do you have a dummy value? Also, any reason why you don’t prefer secrets request?
a
We prefer to configure env and secret in k8s level, not in the workflows (the developers don’t have access to the k8s secret we create)
j
Hmm the pod template looks reasonable as well. Not sure why this would be flaky. It should either work or not. I’ll try to repro.
Flyte and flytekit version @Ariel Kaspit?
a
The dummy value for pod template is because we set the image with the pyflyte command
Flyte chart version: 1.6.2 Flytekit version; 1.7.0
j
I can’t seem to repro. What code is throwing the error? flytekit?
a
I guess it’s the flyteadmin/flytepropeller. We get the error while creating the task (= pod in GKE cluster)
j
115 Views