Tom Theisen
05/12/2023, 7:15 PMextras
?
File "/home/ttheisen/repos/chariot/py/apps/v2-model-recommender/internal/workflow/secrets.py", line 1, in <module>
from flytekit import Secret, current_context
File "/home/ttheisen/.cache/pypoetry/virtualenvs/v2-model-recommender-KgGCWpXS-py3.8/lib/python3.8/site-packages/flytekit/__init__.py", line 187, in <module>
from flytekit.extras import pytorch
File "/home/ttheisen/.cache/pypoetry/virtualenvs/v2-model-recommender-KgGCWpXS-py3.8/lib/python3.8/site-packages/flytekit/extras/pytorch/__init__.py", line 18, in <module>
import torch
File "/home/ttheisen/.cache/pypoetry/virtualenvs/v2-model-recommender-KgGCWpXS-py3.8/lib/python3.8/site-packages/torch/__init__.py", line 228, in <module>
_load_global_deps()
File "/home/ttheisen/.cache/pypoetry/virtualenvs/v2-model-recommender-KgGCWpXS-py3.8/lib/python3.8/site-packages/torch/__init__.py", line 189, in _load_global_deps
_preload_cuda_deps(lib_folder, lib_name)
File "/home/ttheisen/.cache/pypoetry/virtualenvs/v2-model-recommender-KgGCWpXS-py3.8/lib/python3.8/site-packages/torch/__init__.py", line 154, in _preload_cuda_deps
raise ValueError(f"{lib_name} not found in the system path {sys.path}")
ValueError: libcublas.so.*[0-9] not found in the system path ['/home/ttheisen/repos/chariot/py/apps/v2-model-recommender', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/ttheisen/.cache/pypoetry/virtualenvs/v2-model-recommender-KgGCWpXS-py3.8/lib/python3.8/site-packages']
make: *** [Makefile:34: run] Error 1
Kevin Su
05/12/2023, 8:48 PMEduardo Apolinario (eapolinario)
05/12/2023, 9:17 PMBut a ValueError is being raised (not ImportError or OSError)this is the key here. We're being too optimistic in https://github.com/flyteorg/flytekit/blob/master/flytekit/extras/pytorch/__init__.py#L17-L22. For some reason in @Tom Theisen’s case the import statement raises a
ValueError
. We should catch that just to be safe.Tom Theisen
05/12/2023, 9:23 PM