Ena Škopelja
03/14/2023, 9:51 AM{
"json": {
"exec_id": "a98zvdbl4hzdwn95f7xv",
"ns": "development",
"res_ver": "381075629",
"routine": "worker29",
"wf": "flyteexamples:development:container_task_demo.workflows.mmseqs_demo"
},
"level": "error",
"msg": "Error when trying to reconcile workflow. Error [failed at Node[n1]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: [] failed to create resource, caused by: admission webhook \"mutate.kyverno.svc-fail\" denied the request: failed to add image information to the policy rule context: invalid image {{.image.container_task_demo_default.fqn}}:{{.image.container_task_demo_default.version}}]. Error Type[*errors.NodeErrorWithCause]",
"ts": "2023-03-14T09:47:08Z"
}
I registered with the image, here are the logs:
Running pyflyte register from [...] with images ImageConfig(default_image=Image(name='default', fqn='<http://cr.flyte.org/flyteorg/flytekit|cr.flyte.org/flyteorg/flytekit>', tag='py3.9-1.4.1'), images=[Image(name='default', fqn='<http://cr.flyte.org/flyteorg/flytekit|cr.flyte.org/flyteorg/flytekit>', tag='py3.9-1.4.1'), Image(name='container_task_demo_default', fqn='location-docker.pkg.dev/[path to image]', tag='6747b4e2db0d5145e23a00044bdd547d4f276c01')]) and image destination folder /root on 1 package(s)
Registering against flyte.[domain]
Detected Root [...], using this to create deployable package...
Loading packages ['container_task_demo'] under source root [...]
Successfully serialized 1 flyte objects
[✔] Registration container_task_demo.workflows.[task_name] type TASK successful with version 6747b4e2db0d5145e23a00044bdd547d4f276c01
Successfully registered 1 entities
flyte_task
The UI reports the correct image and the task before it runs with that image without issue. I'm not sure what could be causing this.PodSpec
and I'm getting this task config:
{
"config": {
"primary_container_name": "primary"
},
"id": {
"resourceType": 1,
"project": "flyteexamples",
"domain": "development",
"name": "container_task_demo.workflows.task_name",
"version": "6747b4e2db0d5145e23a00044bdd547d4f276c01"
},
"type": "python-task",
"metadata": {
"tags": {
},
"discoverable": true,
"runtime": {
"type": 1,
"version": "1.4.1",
"flavor": "python"
},
"retries": {
},
"discoveryVersion": "0.0.0",
"cacheSerializable": true
},
"interface": {
"inputs": {
"variables": {
"threads": {
"type": {
"unionType": {
"variants": [
{
"structure": {
"tag": "int"
},
"simple": 1
},
{
"structure": {
"tag": "none"
},
"simple": 0
}
]
}
},
"description": "threads"
},
"output": {
"type": {
"blob": {
}
},
"description": "output"
}
}
},
"outputs": {
"variables": {
"o0": {
"type": {
"blob": {
"dimensionality": 1
}
},
"description": "o0"
}
}
}
},
"k8sPod": {
"metadata": # ...,
"podSpec": {
"fields": {
"containers": {
"listValue": {
"values": [
{
"structValue": {
"fields": {
"image": {
"stringValue": "{{.image.container_task_demo_default.fqn}}:{{.image.container_task_demo_default.version}}"
},
"volumeMounts": #...,
"name": # ...
}
}
},
{
"structValue": {
"fields": {
"args": {
"listValue": {
"values": [
{
"stringValue": "pyflyte-execute"
},
{
"stringValue": "--inputs"
},
{
"stringValue": "{{.input}}"
},
{
"stringValue": "--output-prefix"
},
{
"stringValue": "{{.outputPrefix}}"
},
{
"stringValue": "--raw-output-data-prefix"
},
{
"stringValue": "{{.rawOutputDataPrefix}}"
},
{
"stringValue": "--checkpoint-path"
},
{
"stringValue": "{{.checkpointOutputPrefix}}"
},
{
"stringValue": "--prev-checkpoint"
},
{
"stringValue": "{{.prevCheckpointPrefix}}"
},
{
"stringValue": "--resolver"
},
{
"stringValue": "flytekit.core.python_auto_container.default_task_resolver"
},
{
"stringValue": "--"
},
{
"stringValue": "task-module"
},
{
"stringValue": "container_task_demo.workflows"
},
{
"stringValue": "task-name"
},
{
"stringValue": "task name"
}
]
}
},
"env": # ...,
"image": {
"stringValue": "location-docker.pkg.dev/path to image:6747b4e2db0d5145e23a00044bdd547d4f276c01"
},
"name": {
"stringValue": "primary"
},
"command": # ...
}
}
}
]
}
},
"volumes": { #...},
"tolerations": { # ...},
"nodeSelector": { # ... }
}
}
}
}
notice how there's two containers and just one has the correct image resolved
This is how I define the task
task(
cache=True,
cache_version='0.0.0',
pod_template=PodTemplate(
pod_spec=V1PodSpec(containers=[V1Container(name='name',
image='{{.image.container_task_demo_default.fqn}}:'
'{{.image.container_task_demo_default.version}}',
volume_mounts=[...])],
volumes=[...],
node_selector=...,
tolerations=[...,]
),
)):
....
Ketan (kumare3)
03/14/2023, 1:11 PMEna Škopelja
03/14/2023, 1:28 PM