https://flyte.org logo
Join Slack
Powered by
Hi ! We have deployed Flyte in our GCP project an...
# flyte-support
c
Hi ! We have deployed Flyte in our GCP project and we are using Flyte version 1.13.0 at backend/server. Meanwhile we were using 
flytekit==1.11.0
, 
flyteidl==1.11.0
and 
flytekitplugins-pod==1.11.0
packages on local machine to submit the job remotely. Everythig was working fine without any issues. Recently when I update the above mentioned python packages from 
1.11.0
to 
1.13.0
then I am getting the following error when I try to submit the job:
Copy code
FAILED ci/e2e-tests/hello_world/test_hello_world.py::test_workflow_hello_wf - flytekit.exceptions.user.FlyteInvalidInputException: USER:BadInputToAPI: error=None, cause=<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "failed to compile workflow for [resource_type:WORKFLOW project:"playground" domain:"dev" name:"hello.hello_wf" version:"c2d8jj"] with err failed to compile workflow with err Collected Errors: 1
	Error 0: Code: TaskReferenceNotFound, Node Id: start-node, Description: Referenced Task [resource_type:TASK name:"hello.hello_world" version:"c2d8jj"] not found.
"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"failed to compile workflow for [resource_type:WORKFLOW project:\"playground\" domain:\"dev\" name:\"hello.hello_wf\" version:\"c2d8jj\"] with err failed to compile workflow with err Collected Errors: 1\n\tError 0: Code: TaskReferenceNotFound, Node Id: start-node, Description: Referenced Task [resource_type:TASK name:\"hello.hello_world\" version:\"c2d8jj\"] not found.\n", grpc_status:3, created_time:"2024-08-28T14:11:52.293396625+00:00"}"
So, everything is working fine if I am using the python packages of version 
1.11.0
and if I upgrade to 
1.13.0 or 1.13.4
then it didn't work. Are there any compatibility issues with other packages that I should keep in mind to upgrade the flytekit and flyteidl to higher version ? What could be cause of this error ?
v
@crooked-zebra-68589, can you give a few more details? How is the workflow ​`hello.hello­_wf`​ defined?
c
@high-accountant-32689 It is quite simple and basic example. Here is the code:
Copy code
@task()
def hello_world(name: str) -> str:
    """This is a Flyte task that prints 'Hello, World!

    Returns:
        str: "Hello, World!"
    """
    return f"Hello, {name}!"


@workflow
def hello_wf(your_name: str) -> str:
    """This is a Flyte workflow that runs the task!

    Returns:
        str: "Hello, World!"
    """
    return hello_world(name=your_name)
I am submitting the job using the following command:
Copy code
pyflyte run --remote -p playground -d dev hello.py hello_wf --your_name Zeeshan
As I said, everything is working fine if I am using 
flytekit==1.11.0
. When I upgrade to 
flytekit==1.13.0
then I get error.
The following is my complete package list to check any package dependency/conflict:
Copy code
Package                  Version
------------------------ -----------
adlfs                    2024.7.0
aiobotocore              2.13.3
aiohappyeyeballs         2.4.0
aiohttp                  3.10.5
aioitertools             0.11.0
aiosignal                1.3.1
async-timeout            4.0.3
attrs                    24.2.0
azure-core               1.30.2
azure-datalake-store     0.0.53
azure-identity           1.17.1
azure-storage-blob       12.22.0
backports.tarfile        1.2.0
botocore                 1.34.162
cachetools               5.5.0
certifi                  2024.7.4
cffi                     1.17.0
charset-normalizer       3.3.2
click                    8.1.7
cloudpickle              3.0.0
croniter                 3.0.3
cryptography             43.0.0
dataclasses-json         0.5.9
debugpy                  1.8.1
decorator                5.1.1
deflate-dict             1.2.0
dict-hash                1.1.37
diskcache                5.6.3
docker                   7.1.0
docstring_parser         0.16
exceptiongroup           1.2.2
flyteidl                 1.13.2
flytekit                 1.13.4
flytekitplugins-pod      1.13.4
frozenlist               1.4.1
fsspec                   2024.6.1
gcsfs                    2024.6.1
google-api-core          2.19.2
google-auth              2.34.0
google-auth-oauthlib     1.2.1
google-cloud-core        2.4.1
google-cloud-storage     2.18.2
google-crc32c            1.5.0
google-resumable-media   2.7.2
googleapis-common-protos 1.65.0
grpcio                   1.66.0
grpcio-status            1.62.3
hlx-wf                   0.9.0
idna                     3.8
importlib_metadata       8.4.0
iniconfig                2.0.0
isodate                  0.6.1
jaraco.classes           3.4.0
jaraco.context           6.0.1
jaraco.functools         4.0.2
jeepney                  0.8.0
Jinja2                   3.1.4
jmespath                 1.0.1
joblib                   1.4.2
jsonlines                4.0.0
jsonpickle               3.2.2
keyring                  25.3.0
kubernetes               30.1.0
markdown-it-py           3.0.0
MarkupSafe               2.1.5
marshmallow              3.22.0
marshmallow-enum         1.5.1
marshmallow-jsonschema   0.13.0
mashumaro                3.13.1
mdurl                    0.1.2
more-itertools           10.4.0
msal                     1.30.0
msal-extensions          1.2.0
multidict                6.0.5
mypy-extensions          1.0.0
numpy                    1.26.4
oauthlib                 3.2.2
packaging                24.1
pip                      23.0.1
pluggy                   1.5.0
portalocker              2.10.1
proto-plus               1.24.0
protobuf                 4.25.1
protoc-gen-openapiv2     0.0.1
pyasn1                   0.6.0
pyasn1_modules           0.4.0
pycparser                2.22
Pygments                 2.18.0
PyJWT                    2.9.0
pytest                   8.3.2
python-dateutil          2.9.0.post0
python-json-logger       2.0.7
pytimeparse              1.1.8
pytz                     2024.1
PyYAML                   6.0.2
requests                 2.32.3
requests-oauthlib        2.0.0
rich                     13.8.0
rich-click               1.8.3
rsa                      4.9
s3fs                     2024.6.1
SecretStorage            3.3.3
setuptools               65.5.1
six                      1.16.0
statsd                   4.0.1
tomli                    2.0.1
typing_extensions        4.12.2
typing-inspect           0.9.0
urllib3                  2.2.2
websocket-client         1.8.0
wheel                    0.44.0
wrapt                    1.16.0
yarl                     1.9.4
zipp                     3.20.1
h
@crooked-zebra-68589, I'm not able to repro this with the example you gave. I noticed in the logs you shared you're running this from within a test. Is there anything different that you're doing there?
c
When I submit the same workflow from my local machine then everything works fine and the Job is successful. We are running the same workflow using our GitLab CICD pipeline in order to check after every commit that basic workflows are running. In our GitLab pipeline execution, it is getting failed. The 
pip list
that I sent you previously is also from the GitLab CICD. The following is the code to run the same workflow using pytest in GitLab CICD:
Copy code
import os

import pytest
from flytekit.remote.remote import FlyteRemote

@pytest.fixture
def remote_config():
    from flytekit.configuration import Config

    return FlyteRemote(
        Config.auto(FLYTE_CONFIG),
        default_domain="dev",
        default_project="playground",
    )


def test_workflow_hello_wf(remote_config: FlyteRemote):
    import random
    import string
    import sys

    sys.path.append(HELLO_WORLD_PKG)
    import hello
    from flytekit.configuration import ImageConfig

    # Avoid collisions on flyte execution names by adding random suffix.
    # Moreover, this random string makes flyte not resusing former registered workflows.
    random_suffix = "".join(
        random.choice(string.ascii_lowercase + string.digits) for _ in range(6)
    )
    execution_name = (
        f"{test_workflow_hello_wf.__name__.replace('_', '-')}-{random_suffix}"
    )

    # First we have to upload python files we wanna execute.
    #
    # Note: Due to a bug params "domain" and "project" won't get
    # considered. As a workaround we can set "default_domain" and
    # "default_project" on initialization of FlyteRemote().
    remote_config.register_script(
        entity=hello.hello_wf,
        version=random_suffix,
        image_config=ImageConfig.from_images(HLX_TASK_IMAGE),
        source_path=HELLO_WORLD_PKG,
        copy_all=True,
    )

    execution = remote_config.execute(
        entity=hello.hello_wf,
        version=random_suffix,
        inputs={"your_name": "e2e-tester"},
        execution_name=execution_name,
        wait=True,
        image_config=ImageConfig.from_images(HLX_TASK_IMAGE),
    )

    result = execution.outputs.get("o0", as_type=str)

    assert f"Hello, e2e-tester!" == result
On my local machine, if I delete all my previous installations, python and all the packages and then if I directly install 
flytekit==1.13.0
or 
flytekit==1.13.4
then it also didn't work and I get the same error.
a
The issue can be reproduced locally pretty easily:
flytekit_issue.md
flytekit_issue.md
👍 1
Just execute the steps explained in the Markdown file
5 Views
Open in Slack
PreviousNext
Powered by