https://flyte.org logo
#ask-the-community
Title
# ask-the-community
f

Fabio Grätz

12/06/2023, 3:07 PM
Overriding the container image isn’t working for me 🤔
Copy code
@task
def foo() -> str:
    return "foo"


@workflow
def wf():
    foo().with_overrides(
        container_image="{{.image.default.fqn}}:" + "9410d0f0ed4a25577ab35a79bd3eb1119d8627d59c7a8fb947d42ca9fb46a61c"
    )
I can in fact specify any version
"{{.image.default.fqn}}:" + "non-existent"
and the task in the UI still shows the “default image”. I saw that there was very recent work fixing container image overrides for map tasks. Could this be broken also for normal tasks? Or am I using it the wrong way?
Tagging you @Kevin Su because you recently looked at the map tasks issue 🙏
s

Samhita Alla

12/06/2023, 4:54 PM
i just tested it on the demo cluster and it's working fine.
have you tried specifying a hard-coded image? does that work for you?
k

Kevin Su

12/06/2023, 10:49 PM
it’s also working for me.I’m using latest flytekit.
f

Fabio Grätz

12/07/2023, 10:44 AM
Thanks for trying on your end 🙏 I now realized that for us it works if we use
pyflyte run
(which we mostly don’t use) but does not work if we use
pyflyte register
and then run from the UI.
Could you please test whether you see the same?
s

Samhita Alla

12/07/2023, 11:01 AM
yes, i'm able to reproduce. would you mind creating an issue? [flyte-bug]
s

Samhita Alla

12/07/2023, 11:01 AM
cc @Eduardo Apolinario (eapolinario)
f

Fabio Grätz

12/07/2023, 11:12 AM
Yes, thanks!
@Eduardo Apolinario (eapolinario) @Kevin Su do you have an idea what the reason could be? In case you have hints, I would try fixing it since this feat would help us a lot soonish 🙂
k

Kevin Su

12/07/2023, 7:37 PM
I guess flytekit compiles the workflow after it register
foo
1. register foo with default image 2. compile the workflow, and register the tasks in the workflow 3. foo already exists, so flytekit won’t register it again we should compile the workflow first 1. compile the workflow, and register foo with new image in the workflow
f

Fabio Grätz

12/07/2023, 8:15 PM
Wouldn’t this mean that in general
pyflyte register
doesn’t work with overrides because of the order in which things are happening?
k

Kevin Su

12/07/2023, 10:15 PM
container image is special because it’s task level metadata, so if you have two tasks like below,
Copy code
@workflow
def wf():
    foo().with_overrides(container_image="image1")
    foo().with_overrides(container_image="image2")
these two tasks in your workflow will use same image, because flytekit only register one task entity.
people usually override image in the dynamic task since dynamic task will register a new task entity
why can’t you use
Copy code
@task(container_image="{{.image.default.fqn}}:" + "9410d0f0ed4a25577ab35a79bd3eb1119d8627d59c7a8fb947d42ca9fb46a61c")
def foo() -> str:
    return "foo"


@workflow
def wf():
    foo()
f

Fabio Grätz

12/07/2023, 11:25 PM
We need to run an experiment with a set of different versions of an image. We planned to loop over a list of images and use
with_overrides
to run a task with each of them but it sounds like this wouldn’t work because the overriding happens locally before registering. Do you think it would work if we loop over the images, start a dynamic for each and then override the image in there? 🤔
I’ll try tomorrow morning.
Resources are a task level metadata too, right? But for resources the overriding happens in the backend?
k

Kevin Su

12/07/2023, 11:33 PM
TaskNodeOverrides only has resource for now
f

Fabio Grätz

12/08/2023, 3:00 PM
Thanks for the hint to use
with_overrides(container_image=…)
together with dynamic @Kevin Su. With this workaround, the team that asked me about this is unblocked and can do their experiments … 🙂
Generally, I feel it would be good to treat all overrides as task node overrides as currently this feels somewhat broken. I documented the reason for the behaviour and also the workaround with dynamic in the issue.