https://flyte.org
Join Slack
Overriding the container image isn’t working for me :thinking_face: ```@task def foo() -> str: ...
# flyte-support
c
Overriding the container image isn’t working for me 🤔
Copy code
@task
def foo() -> str:
    return "foo"


@workflow
def wf():
    foo().with_overrides(
        container_image="{{.image.default.fqn}}:" + "9410d0f0ed4a25577ab35a79bd3eb1119d8627d59c7a8fb947d42ca9fb46a61c"
    )
I can in fact specify any version 
"{{.image.default.fqn}}:" + "non-existent"
and the task in the UI still shows the “default image”. I saw that there was very recent work fixing container image overrides for map tasks. Could this be broken also for normal tasks? Or am I using it the wrong way?
Tagging you @glamorous-carpet-83516 because you recently looked at the map tasks issue 🙏
t
i just tested it on the demo cluster and it's working fine.
have you tried specifying a hard-coded image? does that work for you?
g
it’s also working for me.I’m using latest flytekit.
c
Thanks for trying on your end 🙏 I now realized that for us it works if we use 
pyflyte run
(which we mostly don’t use) but does not work if we use 
pyflyte register
and then run from the UI.
Could you please test whether you see the same?
t
yes, i'm able to reproduce. would you mind creating an issue? [flyte-bug]
u
t
cc @high-accountant-32689
c
Yes, thanks!
@high-accountant-32689 @glamorous-carpet-83516 do you have an idea what the reason could be? In case you have hints, I would try fixing it since this feat would help us a lot soonish 🙂
g
I guess flytekit compiles the workflow after it register 
foo
1. register foo with default image 2. compile the workflow, and register the tasks in the workflow 3. foo already exists, so flytekit won’t register it again we should compile the workflow first 1. compile the workflow, and register foo with new image in the workflow
c
Wouldn’t this mean that in general 
pyflyte register
doesn’t work with overrides because of the order in which things are happening?
g
container image is special because it’s task level metadata, so if you have two tasks like below,
Copy code
@workflow
def wf():
    foo().with_overrides(container_image="image1")
    foo().with_overrides(container_image="image2")
these two tasks in your workflow will use same image, because flytekit only register one task entity.
people usually override image in the dynamic task since dynamic task will register a new task entity
why can’t you use
Copy code
@task(container_image="{{.image.default.fqn}}:" + "9410d0f0ed4a25577ab35a79bd3eb1119d8627d59c7a8fb947d42ca9fb46a61c")
def foo() -> str:
    return "foo"


@workflow
def wf():
    foo()
c
We need to run an experiment with a set of different versions of an image. We planned to loop over a list of images and use 
with_overrides
to run a task with each of them but it sounds like this wouldn’t work because the overriding happens locally before registering. Do you think it would work if we loop over the images, start a dynamic for each and then override the image in there? 🤔
I’ll try tomorrow morning.
Resources are a task level metadata too, right? But for resources the overriding happens in the backend?
g
TaskNodeOverrides only has resource for now
c
Thanks for the hint to use 
with_overrides(container_image=…)
together with dynamic @glamorous-carpet-83516. With this workaround, the team that asked me about this is unblocked and can do their experiments … 🙂
Generally, I feel it would be good to treat all overrides as task node overrides as currently this feels somewhat broken. I documented the reason for the behaviour and also the workaround with dynamic in the issue.
18 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by