Hi all, I am developing an imperative workflow that is dynamically built based on a json. I am using the union.ai platform for testing. This part I have already developed and it runs correctly if I register the tasks and run the workflow through the command line.

union register clf --project default --domain development
union run --remote --copy-all script_pyflyte.py imperative_wf

However, when I try to run the workflow by connecting from the code with remote it fails with this error. ->

[1/1] currentAttempt done. Last Error: USER::admission webhook "<http://image-builder-webhook.union.ai|image-builder-webhook.union.ai>" denied the request: access to classifiers_spec:Oh6eZo6iI15tuUomnYPN7w is not authorized

remote = UnionRemote(
    config=Config.for_endpoint(endpoint=endpoint),
    default_project=project,
    default_domain=domain,
)

image_config = ImageConfig.auto_default_image()

registered_workflow = remote.register_script(
    outputs,
    copy_all=True,
    default_launch_plan=False,
    image_config=image_config,
    project=project,
    domain=domain,
    source_path=Path(__file__).parent.parent.parent.resolve(),
    version=version,
)

execution = remote.execute_remote_wf(
    registered_workflow, inputs={}, project=project, domain=domain
)

I have tried with various image configurations and also using a built and publicly accessible image but still I get the same error. What can I do to fix this error? I appreciate any help 🙂

Cc @high-park-82026 @acceptable-policeman-57188 can you help

hey @prehistoric-hydrogen-4945 right now public images are not supported in union serverless so unfortunately this is expected

hey @acceptable-policeman-57188, thanks for the answer. I already try update the ImageSpec with my requirements.

image = ImageSpec(
    name="classifiers",
    packages=[
        "accelerate>=0.20.1",
        "adlfs>=2023.4.0",
        "ai2-tango>=1.3.0",
        "azure-storage-blob>=12.17.0",
        "datasets>=2.14.6",
        "docker==6.1.3",
        "einops>=0.6.1",
        "mlflow>=2.12.1",
        "nlpaug",
        "onnx==1.15.0",
        "onnxruntime==1.16.2",
        "onnxruntime-gpu==1.16.2",
        "plotnine",
        "scikit-learn",
        "sentence-transformers>=2.2.2",
        "tabulate",
        "torch>=2.0.1",
        "einops>=0.6.1",
        "accelerate",
        "bitsandbytes",
        "tritonclient[http]==2.39.0",
    ],
)

@task(
    container_image=image, cache=True, cache_version="6",
)
def load_data_from_disk(dataset_path: str, ...):
   ...

I register my tasks using

union register

, then a build_image execution is launched, building my container image. But now when I try to run remote.register_script I get same error ->

[1/1] currentAttempt done. Last Error: USER::admission webhook "<http://image-builder-webhook.union.ai|image-builder-webhook.union.ai>" denied the request: access to classifiers:4V7kczm3iHLfhnAN69NKAA is not authorized.

Hmm this is weird are you using programmatic way of launching ? Cc @acceptable-policeman-57188

hey @prehistoric-hydrogen-4945 where do you set

classifiers:4V7kczm3iHLfhnAN69NKAA

during registration in that case? cc @flaky-parrot-42438

Hi Alvaro, I am happy to jump on a call to walk through your example. I have used imperative (programmatic) workflows before, and they are quite powerful. I am really curious about your use case! Also, while you could explicitly define all of your dependencies, I personally love using the

requirements

parameter of `ImageSpec`:

import flytekit

image = flytekit.ImageSpec(requirements="requirements.txt")

Where you may define

requirements.txt

with a command like this:

uv pip compile pyproject.toml > requirements.txt --python-platform linux

This will freeze all of your local dependencies. This works really well in handling complex version dependencies issues.

hey @acceptable-policeman-57188. Using this command automatically generates the image with the label.

union register clf --project default --domain development

When I launch the imperative workflow it automatically takes the image and label that I had generated when registering the tasks. I have also tried running the

union run

command directly without registering the tasks and it also launches the process to build the image directly. The problem is that if I launch it with union command line it works.

union run --remote --copy-all run_imperative_wf.py outputs

But if I do it using UnionRemote as I showed you in the code I get the

USER::admission

error.

Thank you for the details! I see where the bug is and thinking of a fix.

hey @flaky-parrot-42438 I have tried to generate the image as you said. When I register the tasks, the image is registered correctly. However, when I launch the workflow, the following error is displayed and the execution in union.ai does not appear.

/home/aromo/aromo-union/lib/python3.10/site-packages/flytekit/image_spec/image_spec.py:136 in    │
│ exist                                                                                            │
│                                                                                                  │
│   133 │   │   except Exception as e:                                                             │
│   134 │   │   │   tag = calculate_hash_from_image_spec(self)                                     │
│   135 │   │   │   # if docker engine is not running locally, use requests to check if the imag   │
│ ❱ 136 │   │   │   if "localhost:" in self.registry:                                              │
│   137 │   │   │   │   container_registry = self.registry                                         │
│   138 │   │   │   elif self.registry and "/" in self.registry:                                   │
│   139 │   │   │   │   container_registry = self.registry.split("/")[0]                           │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │      APIError = <class 'docker.errors.APIError'>                                             │ │
│ │        docker = <module 'docker' from                                                        │ │
│ │                 '/home/aromo/aromo-union/lib/python3.10/site-packages/docker/__init__.py'>   │ │
│ │ ImageNotFound = <class 'docker.errors.ImageNotFound'>                                        │ │
│ │          self = ImageSpec(                                                                   │ │
│ │                 │   name='classifiers_v2',                                                   │ │
│ │                 │   python_version=None,                                                     │ │
│ │                 │   builder=None,                                                            │ │
│ │                 │   source_root=None,                                                        │ │
│ │                 │   env=None,                                                                │ │
│ │                 │   registry=None,                                                           │ │
│ │                 │   packages=None,                                                           │ │
│ │                 │   conda_packages=None,                                                     │ │
│ │                 │   conda_channels=None,                                                     │ │
│ │                 │   requirements='requirements_union.txt',                                   │ │
│ │                 │   apt_packages=None,                                                       │ │
│ │                 │   cuda=None,                                                               │ │
│ │                 │   cudnn=None,                                                              │ │
│ │                 │   base_image=None,                                                         │ │
│ │                 │   platform='linux/amd64',                                                  │ │
│ │                 │   pip_index=None,                                                          │ │
│ │                 │   pip_extra_index_url=None,                                                │ │
│ │                 │   registry_config=None,                                                    │ │
│ │                 │   entrypoint=None,                                                         │ │
│ │                 │   commands=None,                                                           │ │
│ │                 │   tag_format=None                                                          │ │
│ │                 )                                                                            │ │
│ │           tag = '3lqAJSS3fFLROiVbqpa1_Q'                                                     │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: argument of type 'NoneType' is not iterable

Going to that branch is unexpected for

registry=None

. I think It should have went to: https://github.com/thomasjpfan/flytekit/blob/eb204594a5f8cd4a662d4bd44ab7e8d7dea22a98/flytekit/image_spec/image_spec.py#L115-L116 CC @glamorous-carpet-83516

hey everyone, any updates on this topic?

Can you share the execution url that failed for you?