https://flyte.org logo
Join Slack
Powered by
I'm getting an error from flytepropeller claiming ...
# flyte-support
s
I'm getting an error from flytepropeller claiming I have a wrong image:
Copy code
{
  "json": {
    "exec_id": "a98zvdbl4hzdwn95f7xv",
    "ns": "development",
    "res_ver": "381075629",
    "routine": "worker29",
    "wf": "flyteexamples:development:container_task_demo.workflows.mmseqs_demo"
  },
  "level": "error",
  "msg": "Error when trying to reconcile workflow. Error [failed at Node[n1]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: [] failed to create resource, caused by: admission webhook \"mutate.kyverno.svc-fail\" denied the request: failed to add image information to the policy rule context: invalid image {{.image.container_task_demo_default.fqn}}:{{.image.container_task_demo_default.version}}]. Error Type[*errors.NodeErrorWithCause]",
  "ts": "2023-03-14T09:47:08Z"
}
I registered with the image, here are the logs:
Copy code
Running pyflyte register from [...] with images ImageConfig(default_image=Image(name='default', fqn='<http://cr.flyte.org/flyteorg/flytekit|cr.flyte.org/flyteorg/flytekit>', tag='py3.9-1.4.1'), images=[Image(name='default', fqn='<http://cr.flyte.org/flyteorg/flytekit|cr.flyte.org/flyteorg/flytekit>', tag='py3.9-1.4.1'), Image(name='container_task_demo_default', fqn='location-docker.pkg.dev/[path to image]', tag='6747b4e2db0d5145e23a00044bdd547d4f276c01')]) and image destination folder /root on 1 package(s) 
Registering against flyte.[domain]
Detected Root [...], using this to create deployable package...
Loading packages ['container_task_demo'] under source root [...]
Successfully serialized 1 flyte objects
[✔] Registration container_task_demo.workflows.[task_name] type TASK successful with version 6747b4e2db0d5145e23a00044bdd547d4f276c01
Successfully registered 1 entities
flyte_task
The UI reports the correct image and the task before it runs with that image without issue. I'm not sure what could be causing this.
Hmm, ok I see where the issue is coming from, I use 
PodSpec
and I'm getting this task config:
Copy code
{
  "config": {
    "primary_container_name": "primary"
  },
  "id": {
    "resourceType": 1,
    "project": "flyteexamples",
    "domain": "development",
    "name": "container_task_demo.workflows.task_name",
    "version": "6747b4e2db0d5145e23a00044bdd547d4f276c01"
  },
  "type": "python-task",
  "metadata": {
    "tags": {
    
    },
    "discoverable": true,
    "runtime": {
      "type": 1,
      "version": "1.4.1",
      "flavor": "python"
    },
    "retries": {
    
    },
    "discoveryVersion": "0.0.0",
    "cacheSerializable": true
  },
  "interface": {
    "inputs": {
      "variables": {
        "threads": {
          "type": {
            "unionType": {
              "variants": [
                {
                  "structure": {
                    "tag": "int"
                  },
                  "simple": 1
                },
                {
                  "structure": {
                    "tag": "none"
                  },
                  "simple": 0
                }
              ]
            }
          },
          "description": "threads"
        },
        "output": {
          "type": {
            "blob": {
            
            }
          },
          "description": "output"
        }
      }
    },
    "outputs": {
      "variables": {
        "o0": {
          "type": {
            "blob": {
              "dimensionality": 1
            }
          },
          "description": "o0"
        }
      }
    }
  },
  "k8sPod": {
    "metadata": # ...,
    "podSpec": {
      "fields": {
        "containers": {
          "listValue": {
            "values": [
              {
                "structValue": {
                  "fields": {
                    "image": {
                      "stringValue": "{{.image.container_task_demo_default.fqn}}:{{.image.container_task_demo_default.version}}"
                    },
                    "volumeMounts": #...,
                    "name": # ... 
                  }
                }
              },
              {
                "structValue": {
                  "fields": {
                    "args": {
                      "listValue": {
                        "values": [
                          {
                            "stringValue": "pyflyte-execute"
                          },
                          {
                            "stringValue": "--inputs"
                          },
                          {
                            "stringValue": "{{.input}}"
                          },
                          {
                            "stringValue": "--output-prefix"
                          },
                          {
                            "stringValue": "{{.outputPrefix}}"
                          },
                          {
                            "stringValue": "--raw-output-data-prefix"
                          },
                          {
                            "stringValue": "{{.rawOutputDataPrefix}}"
                          },
                          {
                            "stringValue": "--checkpoint-path"
                          },
                          {
                            "stringValue": "{{.checkpointOutputPrefix}}"
                          },
                          {
                            "stringValue": "--prev-checkpoint"
                          },
                          {
                            "stringValue": "{{.prevCheckpointPrefix}}"
                          },
                          {
                            "stringValue": "--resolver"
                          },
                          {
                            "stringValue": "flytekit.core.python_auto_container.default_task_resolver"
                          },
                          {
                            "stringValue": "--"
                          },
                          {
                            "stringValue": "task-module"
                          },
                          {
                            "stringValue": "container_task_demo.workflows"
                          },
                          {
                            "stringValue": "task-name"
                          },
                          {
                            "stringValue": "task name"
                          }
                        ]
                      }
                    },
                    "env": # ...,
                    "image": {
                      "stringValue": "location-docker.pkg.dev/path to image:6747b4e2db0d5145e23a00044bdd547d4f276c01"
                    },
                    "name": {
                      "stringValue": "primary"
                    },
                    "command": # ...
                  }
                }
              }
            ]
          }
        },
        "volumes": { #...},
        "tolerations": { # ...},
        "nodeSelector": { # ... }
      }
    }
  }
}
notice how there's two containers and just one has the correct image resolved This is how I define the task
Copy code
task(
    cache=True,
    cache_version='0.0.0',
    pod_template=PodTemplate(
        pod_spec=V1PodSpec(containers=[V1Container(name='name',
                                                   image='{{.image.container_task_demo_default.fqn}}:'
                                                         '{{.image.container_task_demo_default.version}}',
                                                   volume_mounts=[...])],
                           volumes=[...],
                           node_selector=...,
                           tolerations=[...,]
                           ),
    )): 
....
ok got it, the container had to have the name primary
f
Are you using keyverno
s
afaik no, it's installed in the cluster but I'm not using it
157 Views
Open in Slack
PreviousNext
Powered by