https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
k

KS Tarun

09/22/2022, 12:08 PM
Hi, How to change the container image name in the task decorator to fetch it directly from the sandbox.config file ? I've tried as mentioned below:
Copy code
@task(container_image="{{.image.trainer.fqn }}:{{.image.trainer.version}}")
My sandbox.config looks like this:
Copy code
[images]
trainer = <http://ghcr.io/flyteorg/flytecookbook:core-latest|ghcr.io/flyteorg/flytecookbook:core-latest>
predictor = <http://ghcr.io/flyteorg/flytecookbook:pima_diabetes-d4838f0f5e39a21f845a93b9e3375a675bd75eaa|ghcr.io/flyteorg/flytecookbook:pima_diabetes-d4838f0f5e39a21f845a93b9e3375a675bd75eaa>
I'm getting this error:
Copy code
raise AssertionError(f"Image Config with name {name} not hound in the configuration")
AssertionError:Image Config with name trainer not found in the configuration
Any suggestions about this ?
j

James Evers

09/22/2022, 3:55 PM
hey @KS Tarun, i'm having the same problem. What happens when you try to register your workflow with 
pyflyte --config sandbox.config register ....
?
e

Eduardo Apolinario (eapolinario)

09/22/2022, 9:26 PM
@KS Tarun, how are you registering these workflows?
k

KS Tarun

09/23/2022, 4:30 AM
Hi @James Evers & @Eduardo Apolinario (eapolinario) I'm not registering this workflow. I'm running it with 
pyflyte run --remote script.py wf
s

Samhita Alla

09/26/2022, 5:15 AM
@Smriti Satyan, what’s the right way to access images from the config with 
pyflyte run …
command?
s

Smriti Satyan

09/26/2022, 5:30 AM
pyflyte -c sandbox.config
I believe
j

James Evers

09/26/2022, 3:13 PM
when i run this i get the following error:
Copy code
ValueError: Unimplemented, just specify pkgs like folder/files as args at the end of the command
I'm using Python 3.8.10 and flytekit==1.1.3
s

Samhita Alla

09/26/2022, 3:31 PM
@James Evers, could you send the command you ran?
j

James Evers

09/26/2022, 3:33 PM
yep! it was:
Copy code
pyflyte --config ./sandbox.config register -p flytesnacks -d development --version v1.0 src
s

Samhita Alla

09/26/2022, 3:34 PM
The command’s working when config’s not given, right?
j

James Evers

09/26/2022, 3:35 PM
yeah, and to provide the necessary images i add the 
--image <base-image> --image <other-image>
flags
s

Samhita Alla

09/26/2022, 5:01 PM
cc: @Kevin Su @Yee
Pinging the thread again. cc: @Eduardo Apolinario (eapolinario)
k

Kevin Su

09/28/2022, 7:32 AM
use 
config-sandbox.yaml
instead, and use 
--image
to specify image config. Here is an example.
Copy code
pyflyte --config ~/.flyte/config-sandbox.yaml register --image trainer=<http://ghcr.io/flyteorg/flytecookbook:core-latest|ghcr.io/flyteorg/flytecookbook:core-latest> --image predictor=<http://ghcr.io/flyteorg/flytecookbook:core-latest|ghcr.io/flyteorg/flytecookbook:core-latest> containerization/multi_images.py
For pyflyte run
Copy code
pyflyte --config ~/.flyte/config-remote.yaml run --remote  --image trainer=<http://ghcr.io/flyteorg/flytecookbook:core-latest|ghcr.io/flyteorg/flytecookbook:core-latest> --image predictor=<http://ghcr.io/flyteorg/flytecookbook:core-latest|ghcr.io/flyteorg/flytecookbook:core-latest> containerization/multi_images.py my_workflow
j

James Evers

09/28/2022, 1:16 PM
Thanks @Kevin Su
Is there a way to do the same thing without using 
—image <k>=<v>
? I can imagine this getting clunky when there are more images
@Ketan (kumare3) sorry for the ping, but I noticed you mention a related issue on this thread . Any thoughts w/r/t the above?
k

Ketan (kumare3)

09/28/2022, 2:20 PM
I think I know what’s happening. I think the config.yaml And the sandbox.config are conflicting. Can we try pass the image in command line only
Can I ping back when closer to a desk - morning rush with kids. @Samhita Alla / @Eduardo Apolinario (eapolinario) can either of you put the command pyflyte run —image for @James Evers - @KS Tarun
j

James Evers

09/28/2022, 2:22 PM
No rush! Sorry for the bother
k

Ketan (kumare3)

09/28/2022, 2:25 PM
No this is bad that you are not able Yo run something- just that it’s 7:00 am 😛. You can also hop on the office hours and @Haytham Abuelfutuh can help
j

James Evers

09/28/2022, 2:25 PM
Ah great
e

Eduardo Apolinario (eapolinario)

09/28/2022, 6:16 PM
@James Evers, first, let's confirm that you're using the right config file. Run your 
pyflyte run
command again, but this time can you prepend it with 
FLYTE_SDK_LOGGING_LEVEL=10
? This will tell you exactly what config file was loaded (look for a log line that says something along the lines of 
"Using flytectl/YAML config /home/tux/.flyte/config-sandbox.yaml"}
)
j

James Evers

09/28/2022, 6:19 PM
I’m afk but will do asap
@Eduardo Apolinario (eapolinario) sorry, its not 
pyflyte run
thats giving me the issue, but 
pyflyte register
e

Eduardo Apolinario (eapolinario)

09/28/2022, 8:45 PM
@James Evers, the same idea applies. Can you run your exact 
pyflyte register
command but this time setting the 
FLYTE_SDK_LOGGING_LEVEL
env var to 
10
?
j

James Evers

09/28/2022, 8:46 PM
hey eduardo, was just going to post an update. i must have been doing something really dumb like entering the path to the config incorrectly. when i run any 
pyflyte
command with 
pyflyte --config <fully-qualified-path-to-config>
, everything appears to run correctly
e

Eduardo Apolinario (eapolinario)

09/28/2022, 8:47 PM
yeah, an alternative to passing the config file explicitly in every invocation is to set the 
FLYTECTL_CONFIG
env var to point to the config file
j

James Evers

09/28/2022, 8:48 PM
hmm
i thought that those were different kinds of configs
for example, 
FLYTECTL_CONFIG
points to a file 
config-sandbox.yaml
. That config looks like this:
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: localhost:30081
  authType: Pkce
  insecure: true
logger:
  show-source: true
  level: 0%
the config i was passing to 
pyflyte
commands looks like this:
Copy code
[platform]
url = localhost:30081
insecure = True

[sdk]
workflow_packages=flyte.workflows
e

Eduardo Apolinario (eapolinario)

09/28/2022, 8:50 PM
the format is different, but we're favoring yaml going forward. So, you can unify the two, by adding a section to your yaml file like https://github.com/flyteorg/flytekit/blob/master/tests/flytekit/unit/configuration/configs/sample.yaml#L14-L16
it all goes back to which config file is being used
j

James Evers

09/28/2022, 8:51 PM
ah okay great!
so just to be extra explicit; i can avoid passing a lot of 
--image <k> <v>
flags to 
pyflyte
commands by having the image names specified in the config
e

Eduardo Apolinario (eapolinario)

09/28/2022, 8:52 PM
correct
and making sure that 
pyflyte
is using the correct config file
(by either passing it explicitly, or setting the 
FLYTECTL_CONFIG
env var)
j

James Evers

09/28/2022, 8:59 PM
okay perfecto, thanks for the help!
e

Eduardo Apolinario (eapolinario)

09/28/2022, 8:59 PM
of course! Let us know if you need any help.
s

Sanjay Chouhan

09/30/2022, 5:09 AM
https://github.com/flyteorg/flytekit/blob/master/tests/flytekit/unit/configuration/configs/sample.yaml#L14-L16 It works for pyflyte run but does not for pyflyte register It raises AssertionError(f"Image Config with name {name} not found in the configuration")
I am running, 
pyflyte --config config.yaml register test1.py --version 1.0.1
The config.yaml is,
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///##############-16##<http://26454.us-west-1.elb.amazonaws.com:80|26454.us-west-1.elb.amazonaws.com:80>
  authType: Pkce
  insecure: true
logger:
  show-source: true
  level: 0
images:
  trainer: <http://ghcr.io/flyteorg/flytecookbook:core-latest|ghcr.io/flyteorg/flytecookbook:core-latest>
  predictor: moulee31/sample:1.0
The error is,
Copy code
Traceback (most recent call last):
  File "/home/sanjaychouhan/.local/bin/pyflyte", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/sanjaychouhan/.local/lib/python3.8/site-packages/flytekit/clis/sdk_in_container/register.py", line 174, in register
    registerable_entities = load_packages_and_modules(
  File "/home/sanjaychouhan/.local/lib/python3.8/site-packages/flytekit/tools/repo.py", line 225, in load_packages_and_modules
    registrable_entities = serialize(pkgs_and_modules, ss, str(project_root), options)
  File "/home/sanjaychouhan/.local/lib/python3.8/site-packages/flytekit/tools/repo.py", line 54, in serialize
    registrable_entities = get_registrable_entities(ctx, options=options)
  File "/home/sanjaychouhan/.local/lib/python3.8/site-packages/flytekit/tools/serialize_helpers.py", line 75, in get_registrable_entities
    get_serializable(new_api_serializable_entities, ctx.serialization_settings, entity, options=options)
  File "/home/sanjaychouhan/.local/lib/python3.8/site-packages/flytekit/tools/translator.py", line 578, in get_serializable
    cp_entity = get_serializable_task(entity_mapping, settings, entity)
  File "/home/sanjaychouhan/.local/lib/python3.8/site-packages/flytekit/tools/translator.py", line 173, in get_serializable_task
    container = entity.get_container(settings)
  File "/home/sanjaychouhan/.local/lib/python3.8/site-packages/flytekit/core/python_auto_container.py", line 164, in get_container
    image=get_registerable_container_image(self.container_image, settings.image_config),
  File "/home/sanjaychouhan/.local/lib/python3.8/site-packages/flytekit/core/python_auto_container.py", line 235, in get_registerable_container_image
    raise AssertionError(f"Image Config with name {name} not found in the configuration")
AssertionError: Image Config with name trainer not found in the configuration
@Eduardo Apolinario (eapolinario) @James Evers can you please help me?
j

James Evers

09/30/2022, 12:51 PM
I actually never managed to get around this, but the error I got was something else
k

Ketan (kumare3)

09/30/2022, 1:47 PM
This seems like a bug - cc @Yee
We will try it today and if a bug fix it asap
j

James Evers

09/30/2022, 2:52 PM
thanks!
also, the error i was getting from 
pyflyte --config <path/to/config> register
was:
Copy code
ValueError: Unimplemented, just specify pkgs like folder/files as args at the end of the command
e

Eduardo Apolinario (eapolinario)

09/30/2022, 3:53 PM
@Sanjay Chouhan, @James Evers, I confirmed that this is a bug. Fix coming up ASAP.
j

James Evers

09/30/2022, 3:54 PM
excellent!! thank you!
c

Chandramoulee K V

10/06/2022, 5:48 AM
Hi is this bug fixed or should i raise a issue? Any information on this?
s

Samhita Alla

10/06/2022, 5:55 AM
I think the fix isn’t yet merged. @Eduardo Apolinario (eapolinario), when will the fix be available?
e

Eduardo Apolinario (eapolinario)

10/07/2022, 7:37 AM
flytekit v1.2.1 contains this fix. Can you give it a try, @Chandramoulee K V?
c

Chandramoulee K V

10/10/2022, 5:24 AM
It is working fine @Eduardo Apolinario (eapolinario) Thanks for the update!
4 Views
Powered by