Anindya Saha
03/17/2023, 10:27 PM"Error syncing pod, skipping" err="failed to \"StartContainer\" for \"f154ee96e6c1a4fed852-n0-0\" with ImagePullBackOff: \"Back-off pulling image \\\"<http://cr.flyte.org/flyteorg/flytekit:py3.10-1.2.11\\\|cr.flyte.org/flyteorg/flytekit:py3.10-1.2.11\\\>"\"" pod="flytesnacks-development/f154ee96e6c1a4fed852-n0-0" podUID=5952591c-fde8-47bd-a9c4-2885a01915a7
E0317 22:24:01.121900 71 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"f154ee96e6c1a4fed852-n0-0\" with ImagePullBackOff: \"Back-off pulling image \\\"<http://cr.flyte.org/flyteorg/flytekit:py3.10-1.2.11\\\|cr.flyte.org/flyteorg/flytekit:py3.10-1.2.11\\\>"\"" pod="flytesnacks-development/f154ee96e6c1a4fed852-n0-0" podUID=5952591c-fde8-47bd-a9c4-2885a01915a7
How do I resolve this ?
Posted in #ask-the-communityKetan (kumare3)
pip show flytekit
Anindya Saha
03/17/2023, 11:12 PM(flyteorg) asaha@asaha-mbp182 flyteorg % pip show flytekit
Name: flytekit
Version: 1.2.11
Ok. I can downgrade.Eduardo Apolinario (eapolinario)
03/17/2023, 11:46 PMAnindya Saha
03/17/2023, 11:55 PMRequirement already satisfied: wheel<1.0.0,>=0.30.0 in /opt/anaconda3/envs/flyteorg/lib/python3.10/site-packages (from flytekit==1.4.2) (0.38.4)
ERROR: Could not find a version that satisfies the requirement grpcio<2.0,>=1.50.0 (from flytekit) (from versions: 1.41.0rc2, 1.41.0, 1.41.1, 1.42.0rc1, 1.42.0, 1.43.0rc1, 1.43.0, 1.44.0rc1, 1.44.0rc2, 1.44.0, 1.45.0rc1, 1.45.0, 1.46.0rc1, 1.46.0rc2, 1.46.0, 1.46.1, 1.46.3, 1.46.5, 1.47.0rc1, 1.47.0, 1.47.2, 1.48.0rc1, 1.48.0, 1.48.1, 1.48.2, 1.49.0rc1, 1.49.0rc3, 1.49.0, 1.49.1, 1.50.0rc1)
ERROR: No matching distribution found for grpcio<2.0,>=1.50.0
(flyteorg) asaha@asaha-mbp182 flyteorg % pip freeze | grep grpcio
grpcio==1.49.0
grpcio-status==1.48.2
(flyteorg) asaha@asaha-mbp182 flyteorg %
Ketan (kumare3)
Anindya Saha
03/18/2023, 12:32 AMpip install flytekit scikit-learn --index-url <https://pypi.org/simple>
(flyteorg) asaha@asaha-mbp182 flyteorg %
(flyteorg) asaha@asaha-mbp182 flyteorg % pip freeze | grep flytekit
flytekit==1.4.2
flytekitplugins-whylogs==1.4.2
(flyteorg) asaha@asaha-mbp182 flyteorg %
pip install flytekitplugins.whylogs
Jumped to try out https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/flytekit_plugins/whylogs_examples/index.html
K8s logs shows
{"asctime": "2023-03-18 00:29:38,136", "name": "flytekit", "levelname": "WARNING", "message": "FlyteSchema is deprecated, use Structured Dataset instead."}
Matplotlib created a temporary config/cache directory at /tmp/matplotlib-masd6q2b because the default path (/home/flytekit/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
tar: Removing leading `/' from member names
{"asctime": "2023-03-18 00:29:45,494", "name": "flytekit", "levelname": "WARNING", "message": "FlyteSchema is deprecated, use Structured Dataset instead."}
Traceback (most recent call last):
File "/usr/local/bin/pyflyte-execute", line 8, in <module>
sys.exit(execute_task_cmd())
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/flytekit/bin/entrypoint.py", line 476, in execute_task_cmd
_execute_task(
File "/usr/local/lib/python3.10/site-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/flytekit/bin/entrypoint.py", line 348, in _execute_task
_task_def = resolver_obj.load_task(loader_args=resolver_args)
File "/usr/local/lib/python3.10/site-packages/flytekit/core/python_auto_container.py", line 279, in load_task
task_module = importlib.import_module(name=task_module) # type: ignore
File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/root/whylogs_example.py", line 17, in <module>
import whylogs as why
ModuleNotFoundError: No module named 'whylogs'
Traceback (most recent call last):
File "/usr/local/bin/pyflyte-fast-execute", line 8, in <module>
sys.exit(fast_execute_task_cmd())
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/flytekit/bin/entrypoint.py", line 513, in fast_execute_task_cmd
subprocess.run(cmd, check=True)
File "/usr/local/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['pyflyte-execute', '--inputs', '<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-f40b03d9ec1d846548ea/n0/data/inputs.pb>', '--output-prefix', '<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-f40b03d9ec1d846548ea/n0/data/0>', '--raw-output-data-prefix', '<s3://my-s3-bucket/data/vh/f40b03d9ec1d846548ea-n0-0>', '--checkpoint-path', '<s3://my-s3-bucket/data/vh/f40b03d9ec1d846548ea-n0-0/_flytecheckpoints>', '--prev-checkpoint', '""', '--dynamic-addl-distro', '<s3://my-s3-bucket/flytesnacks/development/2TMTB4SERDGT2U2IPM7HIKMSBU======/scriptmode.tar.gz>', '--dynamic-dest-dir', '/root', '--resolver', 'flytekit.core.python_auto_container.default_task_resolver', '--', 'task-module', 'whylogs_example', 'task-name', 'get_target_data']' returned non-zero exit status 1.
Clearly I need to go slow. I believe minio is emulating the s3 path. I guess I need to provide a container_image with whylogs for the whylogs task. Will continue exploring.Ketan (kumare3)
Samhita Alla
Ketan (kumare3)