https://flyte.org logo
Title
a

Anindya Saha

03/17/2023, 10:27 PM
Hi Team, I am following the first example to launch a test workflow, I see the following error in the docker desktop dashboard console:
"Error syncing pod, skipping" err="failed to \"StartContainer\" for \"f154ee96e6c1a4fed852-n0-0\" with ImagePullBackOff: \"Back-off pulling image \\\"<http://cr.flyte.org/flyteorg/flytekit:py3.10-1.2.11\\\|cr.flyte.org/flyteorg/flytekit:py3.10-1.2.11\\\>"\"" pod="flytesnacks-development/f154ee96e6c1a4fed852-n0-0" podUID=5952591c-fde8-47bd-a9c4-2885a01915a7
E0317 22:24:01.121900      71 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"f154ee96e6c1a4fed852-n0-0\" with ImagePullBackOff: \"Back-off pulling image \\\"<http://cr.flyte.org/flyteorg/flytekit:py3.10-1.2.11\\\|cr.flyte.org/flyteorg/flytekit:py3.10-1.2.11\\\>"\"" pod="flytesnacks-development/f154ee96e6c1a4fed852-n0-0" podUID=5952591c-fde8-47bd-a9c4-2885a01915a7
How do I resolve this ? Posted in #ask-the-community
k

Ketan (kumare3)

03/17/2023, 10:50 PM
@Anindya Saha is this the getting started?
can you tell me
pip show flytekit
@Anindya Saha thank you for pointing this - there is indeed a problem in 1.2.11 of flytekit (the docker container was not published)
we are publishing it right now - the github workflow silently failed 😞
cc @Eduardo Apolinario (eapolinario)
a

Anindya Saha

03/17/2023, 11:12 PM
Hi @Ketan (kumare3) It is so great to hear from you.
(flyteorg) asaha@asaha-mbp182 flyteorg % pip show flytekit

Name: flytekit
Version: 1.2.11
Ok. I can downgrade.
e

Eduardo Apolinario (eapolinario)

03/17/2023, 11:46 PM
@Anindya Saha, we just published that image. Can you retry? (No need to downgrade) Also, any reason why you're running that specific version? Curious to know why you're not running any of the 1.3.x or 1.4.x releases.
a

Anindya Saha

03/17/2023, 11:55 PM
Yes, I had that question as well. I have python 3.10 conda env. When I install flytekit it by default install 1.2.11 but I checked in pypi it is alreadt 1.4.x
Not sure why it auto downloaded a lower version.
THis is the reason
Requirement already satisfied: wheel<1.0.0,>=0.30.0 in /opt/anaconda3/envs/flyteorg/lib/python3.10/site-packages (from flytekit==1.4.2) (0.38.4)
ERROR: Could not find a version that satisfies the requirement grpcio<2.0,>=1.50.0 (from flytekit) (from versions: 1.41.0rc2, 1.41.0, 1.41.1, 1.42.0rc1, 1.42.0, 1.43.0rc1, 1.43.0, 1.44.0rc1, 1.44.0rc2, 1.44.0, 1.45.0rc1, 1.45.0, 1.46.0rc1, 1.46.0rc2, 1.46.0, 1.46.1, 1.46.3, 1.46.5, 1.47.0rc1, 1.47.0, 1.47.2, 1.48.0rc1, 1.48.0, 1.48.1, 1.48.2, 1.49.0rc1, 1.49.0rc3, 1.49.0, 1.49.1, 1.50.0rc1)
ERROR: No matching distribution found for grpcio<2.0,>=1.50.0

(flyteorg) asaha@asaha-mbp182 flyteorg % pip freeze | grep grpcio
grpcio==1.49.0
grpcio-status==1.48.2
(flyteorg) asaha@asaha-mbp182 flyteorg %
k

Ketan (kumare3)

03/18/2023, 12:10 AM
are you using conda install or pip?
a

Anindya Saha

03/18/2023, 12:32 AM
Fixed it. Overriding the index-url. Lyft artifactory is points to earlier grpcio
pip install flytekit scikit-learn --index-url <https://pypi.org/simple>
(flyteorg) asaha@asaha-mbp182 flyteorg % 
(flyteorg) asaha@asaha-mbp182 flyteorg % pip freeze | grep flytekit
flytekit==1.4.2
flytekitplugins-whylogs==1.4.2
(flyteorg) asaha@asaha-mbp182 flyteorg %
Installed
pip install flytekitplugins.whylogs
Jumped to try out https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/flytekit_plugins/whylogs_examples/index.html K8s logs shows
{"asctime": "2023-03-18 00:29:38,136", "name": "flytekit", "levelname": "WARNING", "message": "FlyteSchema is deprecated, use Structured Dataset instead."}
Matplotlib created a temporary config/cache directory at /tmp/matplotlib-masd6q2b because the default path (/home/flytekit/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
tar: Removing leading `/' from member names
{"asctime": "2023-03-18 00:29:45,494", "name": "flytekit", "levelname": "WARNING", "message": "FlyteSchema is deprecated, use Structured Dataset instead."}
Traceback (most recent call last):
  File "/usr/local/bin/pyflyte-execute", line 8, in <module>
    sys.exit(execute_task_cmd())
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/flytekit/bin/entrypoint.py", line 476, in execute_task_cmd
    _execute_task(
  File "/usr/local/lib/python3.10/site-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/flytekit/bin/entrypoint.py", line 348, in _execute_task
    _task_def = resolver_obj.load_task(loader_args=resolver_args)
  File "/usr/local/lib/python3.10/site-packages/flytekit/core/python_auto_container.py", line 279, in load_task
    task_module = importlib.import_module(name=task_module)  # type: ignore
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/root/whylogs_example.py", line 17, in <module>
    import whylogs as why
ModuleNotFoundError: No module named 'whylogs'
Traceback (most recent call last):
  File "/usr/local/bin/pyflyte-fast-execute", line 8, in <module>
    sys.exit(fast_execute_task_cmd())
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/flytekit/bin/entrypoint.py", line 513, in fast_execute_task_cmd
    subprocess.run(cmd, check=True)
  File "/usr/local/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['pyflyte-execute', '--inputs', '<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-f40b03d9ec1d846548ea/n0/data/inputs.pb>', '--output-prefix', '<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-f40b03d9ec1d846548ea/n0/data/0>', '--raw-output-data-prefix', '<s3://my-s3-bucket/data/vh/f40b03d9ec1d846548ea-n0-0>', '--checkpoint-path', '<s3://my-s3-bucket/data/vh/f40b03d9ec1d846548ea-n0-0/_flytecheckpoints>', '--prev-checkpoint', '""', '--dynamic-addl-distro', '<s3://my-s3-bucket/flytesnacks/development/2TMTB4SERDGT2U2IPM7HIKMSBU======/scriptmode.tar.gz>', '--dynamic-dest-dir', '/root', '--resolver', 'flytekit.core.python_auto_container.default_task_resolver', '--', 'task-module', 'whylogs_example', 'task-name', 'get_target_data']' returned non-zero exit status 1.
Clearly I need to go slow. I believe minio is emulating the s3 path. I guess I need to provide a container_image with whylogs for the whylogs task. Will continue exploring.
Is there a way I can find out all the pre-built docker containers flyte has ? I can reuse if there is already one for whylogs.
k

Ketan (kumare3)

03/18/2023, 1:11 PM
You are going fast
This is what I was saying cc @Samhita Alla / @Eduardo Apolinario (eapolinario) / @Niels Bantilan (have default container image per plugin that needs the library
s

Samhita Alla

03/20/2023, 5:12 AM
Could you elaborate on what you mean by default container image, @Ketan (kumare3)? Do you mean we need to modify our integration examples to have a container image per task or should we somehow enforce the image when registering the workflow?
k

Ketan (kumare3)

03/20/2023, 2:24 PM
I will create an issue about this today
This is a feature