https://flyte.org
Join Slack
Menu
Home
Archive
Hi, How to change the container image name in the task decorator to fetch it directly from the sand...
# flyte-support
n
Hi, How to change the container image name in the task decorator to fetch it directly from the sandbox.config file ? I've tried as mentioned below:
Copy code
@task(container_image="{{.image.trainer.fqn }}:{{.image.trainer.version}}")
My sandbox.config looks like this:
Copy code
[images]
trainer = <http://ghcr.io/flyteorg/flytecookbook:core-latest|ghcr.io/flyteorg/flytecookbook:core-latest>
predictor = <http://ghcr.io/flyteorg/flytecookbook:pima_diabetes-d4838f0f5e39a21f845a93b9e3375a675bd75eaa|ghcr.io/flyteorg/flytecookbook:pima_diabetes-d4838f0f5e39a21f845a93b9e3375a675bd75eaa>
I'm getting this error:
Copy code
raise AssertionError(f"Image Config with name {name} not hound in the configuration")
AssertionError:Image Config with name trainer not found in the configuration
Any suggestions about this ?
f
hey @nice-judge-43061, i'm having the same problem. What happens when you try to register your workflow with 
pyflyte --config sandbox.config register ....
?
h
@nice-judge-43061, how are you registering these workflows?
n
Hi @few-animal-62717 & @high-accountant-32689 I'm not registering this workflow. I'm running it with 
pyflyte run --remote script.py wf
t
@billowy-sundown-31926, what’s the right way to access images from the config with 
pyflyte run …
command?
b
pyflyte -c sandbox.config
I believe
f
when i run this i get the following error:
Copy code
ValueError: Unimplemented, just specify pkgs like folder/files as args at the end of the command
I'm using Python 3.8.10 and flytekit==1.1.3
t
@few-animal-62717, could you send the command you ran?
f
yep! it was:
Copy code
pyflyte --config ./sandbox.config register -p flytesnacks -d development --version v1.0 src
t
The command’s working when config’s not given, right?
f
yeah, and to provide the necessary images i add the 
--image <base-image> --image <other-image>
flags
t
cc: @glamorous-carpet-83516 @thankful-minister-83577
Pinging the thread again. cc: @high-accountant-32689
🙏 1
g
use 
config-sandbox.yaml
instead, and use 
--image
to specify image config. Here is an example.
Copy code
pyflyte --config ~/.flyte/config-sandbox.yaml register --image trainer=<http://ghcr.io/flyteorg/flytecookbook:core-latest|ghcr.io/flyteorg/flytecookbook:core-latest> --image predictor=<http://ghcr.io/flyteorg/flytecookbook:core-latest|ghcr.io/flyteorg/flytecookbook:core-latest> containerization/multi_images.py
For pyflyte run
Copy code
pyflyte --config ~/.flyte/config-remote.yaml run --remote  --image trainer=<http://ghcr.io/flyteorg/flytecookbook:core-latest|ghcr.io/flyteorg/flytecookbook:core-latest> --image predictor=<http://ghcr.io/flyteorg/flytecookbook:core-latest|ghcr.io/flyteorg/flytecookbook:core-latest> containerization/multi_images.py my_workflow
f
Thanks @glamorous-carpet-83516
Is there a way to do the same thing without using 
—image <k>=<v>
? I can imagine this getting clunky when there are more images
@freezing-airport-6809 sorry for the ping, but I noticed you mention a related issue on this thread . Any thoughts w/r/t the above?
f
I think I know what’s happening. I think the config.yaml And the sandbox.config are conflicting. Can we try pass the image in command line only
Can I ping back when closer to a desk - morning rush with kids. @tall-lock-23197 / @high-accountant-32689 can either of you put the command pyflyte run —image for @few-animal-62717 - @nice-judge-43061
f
No rush! Sorry for the bother
f
No this is bad that you are not able Yo run something- just that it’s 7:00 am 😛. You can also hop on the office hours and @high-park-82026 can help
f
Ah great
h
@few-animal-62717, first, let's confirm that you're using the right config file. Run your 
pyflyte run
command again, but this time can you prepend it with 
FLYTE_SDK_LOGGING_LEVEL=10
? This will tell you exactly what config file was loaded (look for a log line that says something along the lines of 
"Using flytectl/YAML config /home/tux/.flyte/config-sandbox.yaml"}
)
f
I’m afk but will do asap
@high-accountant-32689 sorry, its not 
pyflyte run
thats giving me the issue, but 
pyflyte register
h
@few-animal-62717, the same idea applies. Can you run your exact 
pyflyte register
command but this time setting the 
FLYTE_SDK_LOGGING_LEVEL
env var to 
10
?
f
hey eduardo, was just going to post an update. i must have been doing something really dumb like entering the path to the config incorrectly. when i run any 
pyflyte
command with 
pyflyte --config <fully-qualified-path-to-config>
, everything appears to run correctly
h
yeah, an alternative to passing the config file explicitly in every invocation is to set the 
FLYTECTL_CONFIG
env var to point to the config file
f
hmm
i thought that those were different kinds of configs
for example, 
FLYTECTL_CONFIG
points to a file 
config-sandbox.yaml
. That config looks like this:
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: localhost:30081
  authType: Pkce
  insecure: true
logger:
  show-source: true
  level: 0%
the config i was passing to 
pyflyte
commands looks like this:
Copy code
[platform]
url = localhost:30081
insecure = True

[sdk]
workflow_packages=flyte.workflows
h
the format is different, but we're favoring yaml going forward. So, you can unify the two, by adding a section to your yaml file like https://github.com/flyteorg/flytekit/blob/master/tests/flytekit/unit/configuration/configs/sample.yaml#L14-L16
it all goes back to which config file is being used
f
ah okay great!
so just to be extra explicit; i can avoid passing a lot of 
--image <k> <v>
flags to 
pyflyte
commands by having the image names specified in the config
h
correct
and making sure that 
pyflyte
is using the correct config file
(by either passing it explicitly, or setting the 
FLYTECTL_CONFIG
env var)
f
okay perfecto, thanks for the help!
h
of course! Let us know if you need any help.
👍 1
g
https://github.com/flyteorg/flytekit/blob/master/tests/flytekit/unit/configuration/configs/sample.yaml#L14-L16 It works for pyflyte run but does not for pyflyte register It raises AssertionError(f"Image Config with name {name} not found in the configuration")
I am running, 
pyflyte --config config.yaml register test1.py --version 1.0.1
The config.yaml is,
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///##############-16##<http://26454.us-west-1.elb.amazonaws.com:80|26454.us-west-1.elb.amazonaws.com:80>
  authType: Pkce
  insecure: true
logger:
  show-source: true
  level: 0
images:
  trainer: <http://ghcr.io/flyteorg/flytecookbook:core-latest|ghcr.io/flyteorg/flytecookbook:core-latest>
  predictor: moulee31/sample:1.0
The error is,
Copy code
Traceback (most recent call last):
  File "/home/sanjaychouhan/.local/bin/pyflyte", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/sanjaychouhan/.local/lib/python3.8/site-packages/flytekit/clis/sdk_in_container/register.py", line 174, in register
    registerable_entities = load_packages_and_modules(
  File "/home/sanjaychouhan/.local/lib/python3.8/site-packages/flytekit/tools/repo.py", line 225, in load_packages_and_modules
    registrable_entities = serialize(pkgs_and_modules, ss, str(project_root), options)
  File "/home/sanjaychouhan/.local/lib/python3.8/site-packages/flytekit/tools/repo.py", line 54, in serialize
    registrable_entities = get_registrable_entities(ctx, options=options)
  File "/home/sanjaychouhan/.local/lib/python3.8/site-packages/flytekit/tools/serialize_helpers.py", line 75, in get_registrable_entities
    get_serializable(new_api_serializable_entities, ctx.serialization_settings, entity, options=options)
  File "/home/sanjaychouhan/.local/lib/python3.8/site-packages/flytekit/tools/translator.py", line 578, in get_serializable
    cp_entity = get_serializable_task(entity_mapping, settings, entity)
  File "/home/sanjaychouhan/.local/lib/python3.8/site-packages/flytekit/tools/translator.py", line 173, in get_serializable_task
    container = entity.get_container(settings)
  File "/home/sanjaychouhan/.local/lib/python3.8/site-packages/flytekit/core/python_auto_container.py", line 164, in get_container
    image=get_registerable_container_image(self.container_image, settings.image_config),
  File "/home/sanjaychouhan/.local/lib/python3.8/site-packages/flytekit/core/python_auto_container.py", line 235, in get_registerable_container_image
    raise AssertionError(f"Image Config with name {name} not found in the configuration")
AssertionError: Image Config with name trainer not found in the configuration
@high-accountant-32689 @few-animal-62717 can you please help me?
f
I actually never managed to get around this, but the error I got was something else
f
This seems like a bug - cc @thankful-minister-83577
We will try it today and if a bug fix it asap
f
thanks!
also, the error i was getting from 
pyflyte --config <path/to/config> register
was:
Copy code
ValueError: Unimplemented, just specify pkgs like folder/files as args at the end of the command
h
@gray-magician-44602, @few-animal-62717, I confirmed that this is a bug. Fix coming up ASAP.
f
excellent!! thank you!
h
Hi is this bug fixed or should i raise a issue? Any information on this?
t
I think the fix isn’t yet merged. @high-accountant-32689, when will the fix be available?
h
flytekit v1.2.1 contains this fix. Can you give it a try, @hallowed-plastic-61640?
🏅 1
h
It is working fine @high-accountant-32689 Thanks for the update!
🙏 1
166 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by