Overriding the container image isn’t working for me 🤔

@task
def foo() -> str:
    return "foo"


@workflow
def wf():
    foo().with_overrides(
        container_image="{{.image.default.fqn}}:" + "9410d0f0ed4a25577ab35a79bd3eb1119d8627d59c7a8fb947d42ca9fb46a61c"
    )

I can in fact specify any version

"{{.image.default.fqn}}:" + "non-existent"

and the task in the UI still shows the “default image”. I saw that there was very recent work fixing container image overrides for map tasks. Could this be broken also for normal tasks? Or am I using it the wrong way?

i just tested it on the demo cluster and it's working fine.

it’s also working for me.I’m using latest flytekit.

Thanks for trying on your end 🙏 I now realized that for us it works if we use

pyflyte run

(which we mostly don’t use) but does not work if we use

pyflyte register

and then run from the UI.

yes, i'm able to reproduce. would you mind creating an issue? [flyte-bug]

🐞 Create a new Flyte Bug issue: https://github.com/flyteorg/flyte/issues/new?assignees=&labels=bug%2Cuntriaged&template=bug_report.yaml&title=%5BBUG%5D+

cc @Eduardo Apolinario (eapolinario)

Yes, thanks!

I guess flytekit compiles the workflow after it register

foo

1. register foo with default image 2. compile the workflow, and register the tasks in the workflow 3. foo already exists, so flytekit won’t register it again we should compile the workflow first 1. compile the workflow, and register foo with new image in the workflow

Wouldn’t this mean that in general

pyflyte register

doesn’t work with overrides because of the order in which things are happening?

container image is special because it’s task level metadata, so if you have two tasks like below,

@workflow
def wf():
    foo().with_overrides(container_image="image1")
    foo().with_overrides(container_image="image2")

these two tasks in your workflow will use same image, because flytekit only register one task entity.

We need to run an experiment with a set of different versions of an image. We planned to loop over a list of images and use

with_overrides

to run a task with each of them but it sounds like this wouldn’t work because the overriding happens locally before registering. Do you think it would work if we loop over the images, start a dynamic for each and then override the image in there? 🤔

I think adding image to TaskNodeOverride can fix it. https://github.com/flyteorg/flytekit/blob/4b37b9d17bab09809f4ba0b0310fd50397a24d31/flytekit/tools/translator.py#L453-L456

Thanks for the hint to use

with_overrides(container_image=…)

together with dynamic @Kevin Su. With this workaround, the team that asked me about this is unblocked and can do their experiments … 🙂