https://flyte.org
Join Slack
Hi, I’m trying to register a workflow using the following method. I need one custom image for a task...
# flyte-support
i
Hi, I’m trying to register a workflow using the following method. I need one custom image for a task which I specified below
Copy code
init_image = Image(name="custom", fqn="custom_fqn", tag="custom_tag")
image_cfg = ImageConfig(images=[init_image])
remote_wf = remote.register_script(entity=aeb_sanity_workflow, copy_all=True, version=args.version, image_config=image_cfg, source_path=".")
When I execute it, it gives the error saying 
ERROR - failed to register the workflow: An image is required for PythonAutoContainer tasks
Do we need to specify 
default_image
as well which is an attribute of 
ImageConfig
, if Yes, what would be the value ?
g
are you using flytekit remote?
i
Yes
g
or build an image by yourself
i
I wanted to see if we had a default image. For instance, if I say 
image_config= ImageConfig.auto_default_image()
, it picks default image. But, since, I want to include custom + default both, isn’t there option where default image can be picked. I see we have package 
flytekit.configuration.DefaultImages
but, it doesn’t give of type 
Image
I see, I can probably call 
DefaultImages.default_image()
and convert it into type 
Image
and put as 
default_image
in 
ImageConfig
g
yes, correct
i
I was able to execute after specifying 
default_image
. One more thing, right now I’m passing the custom_image as part of wf registration, and then later executing it. Is there any way, I can pass the custom_image at the time of execution as an input, so that I can set the 
container_image
parameter for a task during execution like below 
@task(container_image="custom-image-passed-as-input-during-execution")
Or any other way to achieve this ?
g
pyflyte run --image …
i
@glamorous-carpet-83516 My usecase is like this. I want to run some tasks which will be executed using a containerTask. I want to specify a config.yaml during the execution which will contain 
imageName, cmd, args, envs
that we need to pass to the container image. I want these inputs to be given to WF in the form of config.yaml file which inturn can be passed to the ContainerTask. In other sense, I would like to register a WF once, and then during each execution, pass the config.yaml file which should determine the imageName, args, cmd, env that will be passed to the container image. How can I pass the inputs that I got from WF to pass to the ContainerTask
Copy code
ContainerTask(
image="",
command=[],
args=[],
environment=[]
I tried setting a global variable, it worked if I hardcode the image name Working >>>> (Picking up custom image)
Copy code
custom_image = "my-custom-image" // set image at compile time

@task(container_image=custom_image,
      cache=True,
      cache_version="1.0",
      ...)
But, if I set the global variable through some task whose value is passed as input during execution, it doesn’t work. It instead picks up the default python flyte image. Tried with making 
setImage
and 
task1
dynamic as well, still it picks default image Not working >>> (picks up default image)
Copy code
custom_image = ""

@dynamic/task
def setImage(data: Dict):
    global custom_image
    // set image at runtime
    custom_image = data['tasks']['initialize_input']['image']
    return

@dynamic/task(container_image=custom_image,
      cache=True,
      cache_version="1.0",
       ...)
def task1(data: Dict):
   ...

@workflow
def wf(data: Dict):
    setImagePromise = setImage(data=data)
    out = task1(data=data)
    setImagePromise >> out
    return
t
Can you try adding 
container_image
to 
with_overrides
?
Copy code
@workflow
def wf(container_image: str):
    t1().with_overrides(container_image=container_image)
I'm not sure if this will work but please give it a try.
i
No, it didn’t work.
using 
with_overrides
, it still uses the default image, I think the issue is we want the 
container_image
value to to be resolved based on the input during WF execution at runtime. I think workflow is using default image as it’s getting 
container_image
as None during compilation. But, that’s just my theory. I tried to make the 
wf
@dynamic to check if that helps, but there I’m hitting another issue that I’m not able to register the workflow using 
register_script
. Scripts run successfully, but it doesn’t show up in the workflows. I guess 
register_script
only for non-dynamic workflows.
t
@glamorous-carpet-83516, 
container_image
has to work, right?
@incalculable-ice-13425, could you share the workflow and the error you're seeing?
i
I’m running it on my localhost machine using sandbox. I’m not seeing error as such during execution, it’s just that ContainerTask/PythonTask is not picking up the image passed as input during wf execution. It’s taking default python image instead. As shared above in psuedocode for both working/non-working scenario, It’s working if I hardcode the global variable.
t
@incalculable-ice-13425, I was referring to the dynamic workflow registration with 
register_script
😅
i
Oops, sorry ! Yeah, for that it didn’t give any error. I’m also catching exceptions but there was no error. It was supposed to be registered but in UI it shows none.
👍 1
39 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by