Frank Shen
11/22/2022, 6:19 PMflytekit.exceptions.scopes.FlyteScopedUserException: Could not find a renderer for <class 'modin.pandas.dataframe.DataFrame'>
...
File ".../flytekit/types/structured/structured_dataset.py", line 699, in to_html
raise NotImplementedError(f"Could not find a renderer for {type(df)} in {self.Renderers}")
NotImplementedError: Could not find a renderer for <class 'modin.pandas.dataframe.DataFrame'> in {<class 'pandas.core.frame.DataFrame'>: <flytekit.deck.renderer.TopFrameRenderer object at 0x
Eduardo Apolinario (eapolinario)
11/22/2022, 6:54 PMFrank Shen
11/22/2022, 6:58 PMEduardo Apolinario (eapolinario)
11/22/2022, 6:59 PMdisable_deck=True
in the task definition
2. update to flytekit 1.2.3 and try again?Frank Shen
11/22/2022, 7:00 PMEduardo Apolinario (eapolinario)
11/22/2022, 7:01 PMKevin Su
11/22/2022, 7:01 PMEduardo Apolinario (eapolinario)
11/22/2022, 7:05 PMFrank Shen
11/22/2022, 7:08 PMSuccessfully uninstalled flytekit-1.2.0
Successfully installed checksumdir-1.2.0 flytekit-0.32.6 flytekitplugins-modin-0.31.0
Eduardo Apolinario (eapolinario)
11/22/2022, 10:03 PMFrank Shen
11/22/2022, 10:06 PMEduardo Apolinario (eapolinario)
11/22/2022, 10:09 PMpip install flytekitplugins-modin==1.2.4 flytekit==1.2.4
Frank Shen
11/22/2022, 10:11 PMThe conflict is caused by:
The user requested flytekit>=1.2.3
flytekitplugins-snowflake 1.1.1 depends on flytekit<1.2.0 and >=1.1.0b0
The user requested flytekit>=1.2.3
flytekitplugins-snowflake 1.1.0 depends on flytekit<1.2.0 and >=1.1.0b0
The user requested flytekit>=1.2.3
flytekitplugins-snowflake 1.0.5 depends on flytekit<1.2.0 and >=1.0.0b3
The user requested flytekit>=1.2.3
flytekitplugins-snowflake 1.0.4 depends on flytekit<1.2.0 and >=1.1.0b0
The user requested flytekit>=1.2.3
Eduardo Apolinario (eapolinario)
11/22/2022, 10:13 PMFrank Shen
11/22/2022, 10:13 PMflytekitplugins-snowflake==1.2.4
Eduardo Apolinario (eapolinario)
11/22/2022, 10:13 PMpip install flytekitplugins-modin==1.2.4 flytekit==1.2.4 flytekitplugins-snowflake==1.2.4
Frank Shen
11/22/2022, 10:14 PMEduardo Apolinario (eapolinario)
11/22/2022, 10:15 PMFrank Shen
11/22/2022, 10:18 PMThe conflict is caused by:
flytekit 1.2.4 depends on pandas<2.0.0 and >=1.0.0
modin 0.17.0 depends on pandas==1.5.1
flytekit 1.2.4 depends on pandas<2.0.0 and >=1.0.0
modin 0.16.2 depends on pandas==1.5.1
flytekit 1.2.4 depends on pandas<2.0.0 and >=1.0.0
modin 0.16.1 depends on pandas==1.5.0
flytekit 1.2.4 depends on pandas<2.0.0 and >=1.0.0
modin 0.16.0 depends on pandas==1.5.0
flytekit==1.2.4
flytekitplugins-snowflake==1.2.4
flytekitplugins-spark==1.2.4
flytekitplugins-modin==1.2.4
xgboost
ray
modin
xgboost_ray
scikit-learn
Eduardo Apolinario (eapolinario)
11/23/2022, 12:12 AMFrank Shen
11/23/2022, 12:21 AMEduardo Apolinario (eapolinario)
11/23/2022, 12:21 AMFrank Shen
11/23/2022, 12:23 AMEduardo Apolinario (eapolinario)
11/23/2022, 12:29 AMCollecting pandas<2.0.0,>=1.0.0
Using cached <https://maven.homebox.com/repository/max-pypi-releases/packages/pandas/1.3.5/pandas-1.3.5-cp37-cp37m-macosx_10_9_x86_64.whl> (11.0 MB)
pandas==1.5.1
in your requirements file?Frank Shen
11/23/2022, 12:41 AMERROR: Ignored the following versions that require a different python version: 1.4.0 Requires-Python >=3.8; 1.4.0rc0 Requires-Python >=3.8; 1.4.1 Requires-Python >=3.8; 1.4.2 Requires-Python >=3.8; 1.4.3 Requires-Python >=3.8; 1.4.4 Requires-Python >=3.8; 1.5.0 Requires-Python >=3.8; 1.5.0rc0 Requires-Python >=3.8; 1.5.1 Requires-Python >=3.8
ERROR: Could not find a version that satisfies the requirement pandas==1.5.1 (from versions: 0.1, 0.2, 0.3.0, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.5.0, 0.6.0, 0.6.1, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.8.0, 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.13.0, 0.13.1, 0.14.0, 0.14.1, 0.15.0, 0.15.1, 0.15.2, 0.16.0, 0.16.1, 0.16.2, 0.17.0, 0.17.1, 0.18.0, 0.18.1, 0.19.0, 0.19.1, 0.19.2, 0.20.0, 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.21.1, 0.22.0, 0.23.0, 0.23.1, 0.23.2, 0.23.3, 0.23.4, 0.24.0, 0.24.1, 0.24.2, 0.25.0, 0.25.1, 0.25.2, 0.25.3, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5, 1.2.0, 1.2.1, 1.2.2, 1.2.3, 1.2.4, 1.2.5, 1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5)
ERROR: No matching distribution found for pandas==1.5.1
Eduardo Apolinario (eapolinario)
11/23/2022, 12:43 AMFrank Shen
11/23/2022, 12:50 AM1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5)
Eduardo Apolinario (eapolinario)
11/23/2022, 12:51 AMFrank Shen
11/23/2022, 5:40 AMEduardo Apolinario (eapolinario)
11/23/2022, 6:07 PMFrank Shen
11/23/2022, 8:25 PM@workflow
def test_wf_train(use_ray: bool = False)
...
pyflyte run tests/test_xgboost.py test_wf_train --use_ray True
Eduardo Apolinario (eapolinario)
11/23/2022, 8:31 PMTrue
Frank Shen
11/23/2022, 8:36 PMpyflyte run tests/test_xgboost.py test_wf_train --use_ray
?Kevin Su
11/23/2022, 8:46 PMuse_ray
, which means the value of use_ray
is True
pyflyte run tests/test_xgboost.py test_wf_train --use_ray
Frank Shen
11/23/2022, 8:48 PMimport flytekitplugins.modin
@task
def preprocess(df: pd.DataFrame,
use_ray: bool
) -> Union[pd.DataFrame, modin_pd.DataFrame]:
if use_ray:
ray.init()
df = modin.pandas.DataFrame(df)
....
{"asctime": "2022-11-23 13:10:46,651", "name": "flytekit", "levelname": "ERROR", "message": "Failed to convert return value for var o0 with error <class 'TypeError'>: Ambiguous choice of variant for union type"}
Traceback (most recent call last):
File "/Users/fshen/.pyenv/versions/3.8.7/lib/python3.8/site-packages/flytekit/core/base_task.py", line 522, in dispatch_execute
literals[k] = TypeEngine.to_literal(exec_ctx, v, py_type, literal_type)
File "/Users/fshen/.pyenv/versions/3.8.7/lib/python3.8/site-packages/flytekit/core/type_engine.py", line 752, in to_literal
lv = transformer.to_literal(ctx, python_val, python_type, expected)
File "/Users/fshen/.pyenv/versions/3.8.7/lib/python3.8/site-packages/flytekit/core/type_engine.py", line 1060, in to_literal
raise TypeError("Ambiguous choice of variant for union type")
TypeError: Ambiguous choice of variant for union type
Eduardo Apolinario (eapolinario)
11/23/2022, 9:35 PMKevin Su
11/24/2022, 9:59 PM