Hi I m trying to register a workflow using the following met Flyte #ask-the-community

Hi, I’m trying to register a workflow using the fo...

Gaurav Kumar

08/04/2023, 3:47 PM

Hi, I’m trying to register a workflow using the following method. I need one custom image for a task which I specified below

Copy code

init_image = Image(name="custom", fqn="custom_fqn", tag="custom_tag")
image_cfg = ImageConfig(images=[init_image])
remote_wf = remote.register_script(entity=aeb_sanity_workflow, copy_all=True, version=args.version, image_config=image_cfg, source_path=".")

When I execute it, it gives the error saying

ERROR - failed to register the workflow: An image is required for PythonAutoContainer tasks

Do we need to specify

default_image

as well which is an attribute of

ImageConfig

, if Yes, what would be the value ?

Kevin Su

08/04/2023, 4:15 PM

are you using flytekit remote?

Gaurav Kumar

08/04/2023, 4:15 PM

Yes

Kevin Su

08/04/2023, 4:15 PM

you can use default flytekit image. https://github.com/flyteorg/flytekit/pkgs/container/flytekit/115576335?tag=py3.9-1.8.3

Kevin Su

08/04/2023, 4:15 PM

or build an image by yourself

Gaurav Kumar

08/04/2023, 4:21 PM

I wanted to see if we had a default image. For instance, if I say

image_config= ImageConfig.auto_default_image()

, it picks default image. But, since, I want to include custom + default both, isn’t there option where default image can be picked. I see we have package

flytekit.configuration.DefaultImages

but, it doesn’t give of type

Image

Gaurav Kumar

08/04/2023, 4:24 PM

I see, I can probably call

DefaultImages.default_image()

and convert it into type

Image

and put as

default_image

ImageConfig

Kevin Su

08/04/2023, 4:24 PM

yes, correct

Gaurav Kumar

08/04/2023, 5:01 PM

I was able to execute after specifying

default_image

. One more thing, right now I’m passing the custom_image as part of wf registration, and then later executing it. Is there any way, I can pass the custom_image at the time of execution as an input, so that I can set the

container_image

parameter for a task during execution like below

@task(container_image="custom-image-passed-as-input-during-execution")

Or any other way to achieve this ?

Kevin Su

08/04/2023, 9:15 PM

pyflyte run --image …

Gaurav Kumar

08/07/2023, 6:00 AM

@Kevin Su My usecase is like this. I want to run some tasks which will be executed using a containerTask. I want to specify a config.yaml during the execution which will contain

imageName, cmd, args, envs

that we need to pass to the container image. I want these inputs to be given to WF in the form of config.yaml file which inturn can be passed to the ContainerTask. In other sense, I would like to register a WF once, and then during each execution, pass the config.yaml file which should determine the imageName, args, cmd, env that will be passed to the container image. How can I pass the inputs that I got from WF to pass to the ContainerTask

Copy code

ContainerTask(
image="",
command=[],
args=[],
environment=[]

Gaurav Kumar

08/07/2023, 7:53 AM

I tried setting a global variable, it worked if I hardcode the image name Working >>>> (Picking up custom image)

Copy code

custom_image = "my-custom-image" // set image at compile time

@task(container_image=custom_image,
      cache=True,
      cache_version="1.0",
      ...)

But, if I set the global variable through some task whose value is passed as input during execution, it doesn’t work. It instead picks up the default python flyte image. Tried with making

setImage

and

task1

dynamic as well, still it picks default image Not working >>> (picks up default image)

Copy code

custom_image = ""

@dynamic/task
def setImage(data: Dict):
    global custom_image
    // set image at runtime
    custom_image = data['tasks']['initialize_input']['image']
    return

@dynamic/task(container_image=custom_image,
      cache=True,
      cache_version="1.0",
       ...)
def task1(data: Dict):
   ...

@workflow
def wf(data: Dict):
    setImagePromise = setImage(data=data)
    out = task1(data=data)
    setImagePromise >> out
    return

Samhita Alla

08/07/2023, 8:55 AM

Can you try adding

container_image

with_overrides

Copy code

@workflow
def wf(container_image: str):
    t1().with_overrides(container_image=container_image)

I'm not sure if this will work but please give it a try.

Gaurav Kumar

08/07/2023, 9:06 AM

No, it didn’t work.

Gaurav Kumar

08/07/2023, 9:21 AM

using

with_overrides

, it still uses the default image, I think the issue is we want the

container_image

value to to be resolved based on the input during WF execution at runtime. I think workflow is using default image as it’s getting

container_image

as None during compilation. But, that’s just my theory. I tried to make the

wf

@dynamic to check if that helps, but there I’m hitting another issue that I’m not able to register the workflow using

register_script

. Scripts run successfully, but it doesn’t show up in the workflows. I guess

register_script

only for non-dynamic workflows.

Samhita Alla

08/07/2023, 12:53 PM

@Kevin Su,

container_image

has to work, right?

Samhita Alla

08/07/2023, 12:53 PM

@Gaurav Kumar, could you share the workflow and the error you're seeing?

Gaurav Kumar

08/07/2023, 12:58 PM

I’m running it on my localhost machine using sandbox. I’m not seeing error as such during execution, it’s just that ContainerTask/PythonTask is not picking up the image passed as input during wf execution. It’s taking default python image instead. As shared above in psuedocode for both working/non-working scenario, It’s working if I hardcode the global variable.

Samhita Alla

08/07/2023, 3:06 PM

@Gaurav Kumar, I was referring to the dynamic workflow registration with

register_script

😅

Gaurav Kumar

08/07/2023, 4:34 PM

Oops, sorry ! Yeah, for that it didn’t give any error. I’m also catching exceptions but there was no error. It was supposed to be registered but in UI it shows none.

10 Views

Open in Slack

Previous Next