https://flyte.org logo
Powered by
p

Padma Priya M

09/28/2022, 9:36 AM
Hi I am trying to integrate jupyter notebook with flyte. When I execute the script in local I am able to get the output but when I execute in remote I am getting error.
Copy code
[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[ff43e19ed1617402a8c6-n0-0] terminated with exit code (1). Reason [Error]. Message: 
ine 286, in _handle_annotated_task
    _dispatch_execute(ctx, task_def, inputs, output_prefix)
  File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 76, in _dispatch_execute
    logger.debug(f"Starting _dispatch_execute for {task_def.name}")
AttributeError: 'function' object has no attribute 'name'
Traceback (most recent call last):
  File "/opt/venv/bin/pyflyte-fast-execute", line 8, in <module>
    sys.exit(fast_execute_task_cmd())
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1130, in _call_
    return self.main(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 502, in fast_execute_task_cmd
    subprocess.run(cmd, check=True)
  File "/usr/local/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['pyflyte-execute', '--inputs', '<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ff43e19ed1617402a8c6/n0/data/inputs.pb>', '--output-prefix', '<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ff43e19ed1617402a8c6/n0/data/0>', '--raw-output-data-prefix', '<s3://my-s3-bucket/qf/ff43e19ed1617402a8c6-n0-0>', '--checkpoint-path', '<s3://my-s3-bucket/qf/ff43e19ed1617402a8c6-n0-0/_flytecheckpoints>', '--prev-checkpoint', '""', '--dynamic-addl-distro', '<s3://my-s3-bucket/72/flytesnacks/development/LXFFQS6IOSM2OAH57AOZKM2SZA======/scriptmode.tar.gz>', '--dynamic-dest-dir', '/root', '--resolver', 'flytekit.core.python_auto_container.default_task_resolver', '--', 'task-module', 'flytekitplugins.papermill.task', 'task-name', '_dummy_task_func']' returned non-zero exit status 1.
pyflyte run --remote --image ghcr.io/flyteorg/flytecookbook:papermilltasks-latest example.py nb_to_python_wf --f 2.3 command used to execute.
Copy code
import math
import os
import sys
import pathlib

from flytekit import kwtypes, task, workflow
from flytekitplugins.papermill import NotebookTask

nb = NotebookTask(
    name="simple-nb",
    notebook_path=os.path.join(
        sys.path[0], "notebook_sample.ipynb"
    ),
    render_deck=True,
    inputs=kwtypes(v=float),
    outputs=kwtypes(square=float),
)


@task
def square_root_task(f: float) -> float:
    return math.sqrt(f)


@workflow
def nb_to_python_wf(f: float) -> float:
    out = nb(v=f)
    return square_root_task(f=out.square)

if __name__ == "__main__":
    print(nb_to_python_wf(f=3.14))
k

Kevin Su

09/28/2022, 11:19 AM
which flytekit version you are using?
p

Padma Priya M

09/28/2022, 11:25 AM
using flytekit 1.1.3
s

Samhita Alla

09/29/2022, 12:48 PM
This seems like a bug.
@Kevin Su, any idea why this might have cropped up?
k

Kevin Su

09/29/2022, 3:12 PM
yes, this is bug now in 1.1.3. try to downgrade to 1.1.1. I’ll create pr to fix it shortly. btw, currently pyflyte run will only upload the workflow code (example.py) to s3, and will not upload notebook_sample.ipynb. Therefore, you should use pyflyte register instead, and manually trigger workflow by yourself. 
pyflyte register --image <http://ghcr.io/flyteorg/flytecookbook:papermilltasks-latest|ghcr.io/flyteorg/flytecookbook:papermilltasks-latest> example.py
s

Samhita Alla

10/05/2022, 1:42 PM
Have you installed the plugin?
Please install all the requirements in your local environment.
Can you try running the example with the latest flytekit version — 1.2.0? Also, this error is related to not installing the papermill plugin. I’d recommend you to create a new virtual env, update the dependencies (you can copy the requirements from flytesnacks papermill directory and update them using 
pip-compile <http://requirements.in|requirements.in>
), install the dependencies, and run the pyflyte register again.
p

Padma Priya M

10/19/2022, 4:23 AM
Hi, I installed all the dependencies that were mentioned in flytesnacks papermill directory with latest flytekit version. I was able to register the workflow but when I created an execution, I am getting error.
Copy code
#
# This file is autogenerated by pip-compile with python 3.9
# To update, run:
#
#    pip-compile <http://requirements.in|requirements.in>
#
--index-url <https://sonatype.adfdata.net/repository/pypi_adf/simple>
--extra-index-url <https://pypi.org/simple>
--trusted-host <http://sonatype.adfdata.net|sonatype.adfdata.net>

ansiwrap==0.8.4
    # via papermill
arrow==1.2.2
    # via jinja2-time
asttokens==2.0.5
    # via stack-data
attrs==21.4.0
    # via jsonschema
backcall==0.2.0
    # via ipython
beautifulsoup4==4.11.1
    # via nbconvert
binaryornot==0.4.4
    # via cookiecutter
bleach==5.0.0
    # via nbconvert
certifi==2021.10.8
    # via requests
cffi==1.15.1
    # via cryptography
chardet==4.0.0
    # via binaryornot
charset-normalizer==2.0.12
    # via requests
checksumdir==1.2.0
    # via flytekit
click==8.1.2
    # via
    #   cookiecutter
    #   flytekit
    #   papermill
cloudpickle==2.0.0
    # via flytekit
cookiecutter==1.7.3
    # via flytekit
croniter==1.3.4
    # via flytekit
cryptography==38.0.1
    # via secretstorage
dataclasses-json==0.5.7
    # via flytekit
debugpy==1.6.0
    # via ipykernel
decorator==5.1.1
    # via
    #   ipython
    #   retry
defusedxml==0.7.1
    # via nbconvert
deprecated==1.2.13
    # via flytekit
diskcache==5.4.0
    # via flytekit
docker-image-py==0.1.12
    # via flytekit
docstring-parser==0.13
    # via flytekit
entrypoints==0.4
    # via
    #   jupyter-client
    #   nbconvert
    #   papermill
executing==0.8.3
    # via stack-data
fastjsonschema==2.15.3
    # via nbformat
flyteidl==0.24.21
    # via flytekit
flytekit==0.32.3
    # via flytekitplugins-papermill
flytekitplugins-papermill==0.32.3
    # via -r <http://requirements.in|requirements.in>
googleapis-common-protos==1.56.0
    # via
    #   flyteidl
    #   grpcio-status
grpcio==1.44.0
    # via
    #   flytekit
    #   grpcio-status
grpcio-status==1.44.0
    # via flytekit
idna==3.3
    # via requests
importlib-metadata==4.11.3
    # via keyring
ipykernel==6.13.0
    # via flytekitplugins-papermill
ipython==8.2.0
    # via ipykernel
jedi==0.18.1
    # via ipython
jeepney==0.8.0
    # via
    #   keyring
    #   secretstorage
jinja2==3.1.1
    # via
    #   cookiecutter
    #   jinja2-time
    #   nbconvert
jinja2-time==0.2.0
    # via cookiecutter
jsonschema==4.4.0
    # via nbformat
jupyter-client==7.2.2
    # via
    #   ipykernel
    #   nbclient
jupyter-core==4.9.2
    # via
    #   jupyter-client
    #   nbconvert
    #   nbformat
jupyterlab-pygments==0.2.1
    # via nbconvert
keyring==23.5.0
    # via flytekit
markupsafe==2.1.1
    # via
    #   jinja2
    #   nbconvert
marshmallow==3.15.0
    # via
    #   dataclasses-json
    #   marshmallow-enum
    #   marshmallow-jsonschema
marshmallow-enum==1.5.1
    # via dataclasses-json
marshmallow-jsonschema==0.13.0
    # via flytekit
matplotlib-inline==0.1.3
    # via
    #   ipykernel
    #   ipython
mistune==0.8.4
    # via nbconvert
mypy-extensions==0.4.3
    # via typing-inspect
natsort==8.1.0
    # via flytekit
nbclient==0.6.0
    # via
    #   nbconvert
    #   papermill
nbconvert==6.5.0
    # via flytekitplugins-papermill
nbformat==5.3.0
    # via
    #   nbclient
    #   nbconvert
    #   papermill
nest-asyncio==1.5.5
    # via
    #   ipykernel
    #   jupyter-client
    #   nbclient
numpy==1.22.3
    # via
    #   pandas
    #   pyarrow
packaging==21.3
    # via
    #   ipykernel
    #   marshmallow
    #   nbconvert
pandas==1.4.2
    # via flytekit
pandocfilters==1.5.0
    # via nbconvert
papermill==2.3.4
    # via flytekitplugins-papermill
parso==0.8.3
    # via jedi
pexpect==4.8.0
    # via ipython
pickleshare==0.7.5
    # via ipython
poyo==0.5.0
    # via cookiecutter
prompt-toolkit==3.0.29
    # via ipython
protobuf==3.20.0
    # via
    #   flyteidl
    #   flytekit
    #   googleapis-common-protos
    #   grpcio-status
    #   protoc-gen-swagger
protoc-gen-swagger==0.1.0
    # via flyteidl
psutil==5.9.0
    # via ipykernel
ptyprocess==0.7.0
    # via pexpect
pure-eval==0.2.2
    # via stack-data
py==1.11.0
    # via retry
pyarrow==6.0.1
    # via flytekit
pycparser==2.21
    # via cffi
pygments==2.11.2
    # via
    #   ipython
    #   nbconvert
pyparsing==3.0.8
    # via packaging
pyrsistent==0.18.1
    # via jsonschema
python-dateutil==2.8.2
    # via
    #   arrow
    #   croniter
    #   flytekit
    #   jupyter-client
    #   pandas
python-json-logger==2.0.2
    # via flytekit
python-slugify==6.1.1
    # via cookiecutter
pytimeparse==1.1.8
    # via flytekit
pytz==2022.1
    # via
    #   flytekit
    #   pandas
pyyaml==6.0
    # via
    #   flytekit
    #   papermill
pyzmq==22.3.0
    # via jupyter-client
regex==2022.3.15
    # via docker-image-py
requests==2.27.1
    # via
    #   cookiecutter
    #   flytekit
    #   papermill
    #   responses
responses==0.20.0
    # via flytekit
retry==0.9.2
    # via flytekit
secretstorage==3.3.3
    # via keyring
six==1.16.0
    # via
    #   asttokens
    #   bleach
    #   cookiecutter
    #   grpcio
    #   python-dateutil
sortedcontainers==2.4.0
    # via flytekit
soupsieve==2.3.2
    # via beautifulsoup4
stack-data==0.2.0
    # via ipython
statsd==3.3.0
    # via flytekit
tenacity==8.0.1
    # via papermill
text-unidecode==1.3
    # via python-slugify
textwrap3==0.9.2
    # via ansiwrap
tinycss2==1.1.1
    # via nbconvert
tornado==6.1
    # via
    #   ipykernel
    #   jupyter-client
tqdm==4.64.0
    # via papermill
traitlets==5.1.1
    # via
    #   ipykernel
    #   ipython
    #   jupyter-client
    #   jupyter-core
    #   matplotlib-inline
    #   nbclient
    #   nbconvert
    #   nbformat
typing-extensions==4.1.1
    # via
    #   flytekit
    #   typing-inspect
typing-inspect==0.7.1
    # via dataclasses-json
urllib3==1.26.9
    # via
    #   flytekit
    #   requests
    #   responses
wcwidth==0.2.5
    # via prompt-toolkit
webencodings==0.5.1
    # via
    #   bleach
    #   tinycss2
wheel==0.37.1
    # via flytekit
wrapt==1.14.0
    # via
    #   deprecated
    #   flytekit
zipp==3.8.0
    # via importlib-metadata

# The following packages are considered to be unsafe in a requirements file:
# setuptools
@Samhita Alla @Kevin Su
s

Samhita Alla

10/19/2022, 11:43 AM
Your flytekit version is 0.32.3; it should be the latest.
Let me share with you the requirements.txt file.
Copy code
#
# This file is autogenerated by pip-compile with python 3.9
# To update, run:
#
#    /Library/Developer/CommandLineTools/usr/bin/make requirements.txt
#
ansiwrap==0.8.4
    # via papermill
appnope==0.1.3
    # via
    #   ipykernel
    #   ipython
arrow==1.2.3
    # via jinja2-time
asttokens==2.0.8
    # via stack-data
attrs==22.1.0
    # via
    #   jsonschema
    #   visions
backcall==0.2.0
    # via ipython
beautifulsoup4==4.11.1
    # via nbconvert
binaryornot==0.4.4
    # via cookiecutter
bleach==5.0.1
    # via nbconvert
certifi==2022.9.24
    # via requests
cffi==1.15.1
    # via cryptography
chardet==5.0.0
    # via binaryornot
charset-normalizer==2.1.1
    # via requests
click==8.1.3
    # via
    #   cookiecutter
    #   flytekit
    #   papermill
cloudpickle==2.2.0
    # via flytekit
cookiecutter==2.1.1
    # via flytekit
croniter==1.3.7
    # via flytekit
cryptography==38.0.1
    # via pyopenssl
cycler==0.11.0
    # via matplotlib
dataclasses-json==0.5.7
    # via flytekit
debugpy==1.6.3
    # via ipykernel
decorator==5.1.1
    # via
    #   ipython
    #   retry
defusedxml==0.7.1
    # via nbconvert
deprecated==1.2.13
    # via flytekit
diskcache==5.4.0
    # via flytekit
docker==6.0.0
    # via flytekit
docker-image-py==0.1.12
    # via flytekit
docstring-parser==0.15
    # via flytekit
entrypoints==0.4
    # via
    #   jupyter-client
    #   papermill
executing==1.1.1
    # via stack-data
fastjsonschema==2.16.2
    # via nbformat
flyteidl==1.1.22
    # via flytekit
flytekit==1.2.1
    # via
    #   -r ../../../common/requirements-common.in
    #   flytekitplugins-deck-standard
    #   flytekitplugins-papermill
flytekitplugins-deck-standard==1.2.1
    # via -r ../../../common/requirements-common.in
flytekitplugins-papermill==1.2.1
    # via -r <http://requirements.in|requirements.in>
fonttools==4.37.4
    # via matplotlib
googleapis-common-protos==1.56.4
    # via
    #   flyteidl
    #   grpcio-status
grpcio==1.50.0
    # via
    #   flytekit
    #   grpcio-status
grpcio-status==1.48.0
    # via
    #   -r <http://requirements.in|requirements.in>
    #   flytekit
htmlmin==0.1.12
    # via pandas-profiling
idna==3.4
    # via requests
imagehash==4.3.1
    # via visions
importlib-metadata==5.0.0
    # via
    #   flytekit
    #   keyring
    #   markdown
    #   nbconvert
ipykernel==6.16.0
    # via flytekitplugins-papermill
ipython==8.5.0
    # via ipykernel
jaraco-classes==3.2.3
    # via keyring
jedi==0.18.1
    # via ipython
jinja2==3.1.2
    # via
    #   cookiecutter
    #   jinja2-time
    #   nbconvert
    #   pandas-profiling
jinja2-time==0.2.0
    # via cookiecutter
joblib==1.1.1
    # via
    #   flytekit
    #   pandas-profiling
    #   phik
jsonschema==4.16.0
    # via nbformat
jupyter-client==7.4.2
    # via
    #   ipykernel
    #   nbclient
jupyter-core==4.11.1
    # via
    #   jupyter-client
    #   nbconvert
    #   nbformat
jupyterlab-pygments==0.2.2
    # via nbconvert
keyring==23.9.3
    # via flytekit
kiwisolver==1.4.4
    # via matplotlib
markdown==3.4.1
    # via flytekitplugins-deck-standard
markupsafe==2.1.1
    # via
    #   jinja2
    #   nbconvert
marshmallow==3.18.0
    # via
    #   dataclasses-json
    #   marshmallow-enum
    #   marshmallow-jsonschema
marshmallow-enum==1.5.1
    # via dataclasses-json
marshmallow-jsonschema==0.13.0
    # via flytekit
matplotlib==3.5.3
    # via
    #   -r ../../../common/requirements-common.in
    #   missingno
    #   pandas-profiling
    #   phik
    #   seaborn
matplotlib-inline==0.1.6
    # via
    #   ipykernel
    #   ipython
missingno==0.5.1
    # via pandas-profiling
mistune==2.0.4
    # via nbconvert
more-itertools==9.0.0
    # via jaraco-classes
multimethod==1.8
    # via
    #   pandas-profiling
    #   visions
mypy-extensions==0.4.3
    # via typing-inspect
natsort==8.2.0
    # via flytekit
nbclient==0.7.0
    # via
    #   nbconvert
    #   papermill
nbconvert==7.2.1
    # via flytekitplugins-papermill
nbformat==5.7.0
    # via
    #   nbclient
    #   nbconvert
    #   papermill
nest-asyncio==1.5.6
    # via
    #   ipykernel
    #   jupyter-client
    #   nbclient
networkx==2.8.7
    # via visions
numpy==1.23.4
    # via
    #   imagehash
    #   matplotlib
    #   missingno
    #   pandas
    #   pandas-profiling
    #   patsy
    #   phik
    #   pyarrow
    #   pywavelets
    #   scipy
    #   seaborn
    #   statsmodels
    #   visions
packaging==21.3
    # via
    #   docker
    #   ipykernel
    #   marshmallow
    #   matplotlib
    #   nbconvert
    #   statsmodels
pandas==1.4.4
    # via
    #   flytekit
    #   pandas-profiling
    #   phik
    #   seaborn
    #   statsmodels
    #   visions
pandas-profiling==3.3.0
    # via flytekitplugins-deck-standard
pandocfilters==1.5.0
    # via nbconvert
papermill==2.4.0
    # via flytekitplugins-papermill
parso==0.8.3
    # via jedi
patsy==0.5.3
    # via statsmodels
pexpect==4.8.0
    # via ipython
phik==0.12.2
    # via pandas-profiling
pickleshare==0.7.5
    # via ipython
pillow==9.2.0
    # via
    #   imagehash
    #   matplotlib
    #   visions
plotly==5.10.0
    # via flytekitplugins-deck-standard
prompt-toolkit==3.0.31
    # via ipython
protobuf==3.20.3
    # via
    #   flyteidl
    #   flytekit
    #   googleapis-common-protos
    #   grpcio-status
    #   protoc-gen-swagger
protoc-gen-swagger==0.1.0
    # via flyteidl
psutil==5.9.3
    # via ipykernel
ptyprocess==0.7.0
    # via pexpect
pure-eval==0.2.2
    # via stack-data
py==1.11.0
    # via retry
pyarrow==6.0.1
    # via flytekit
pycparser==2.21
    # via cffi
pydantic==1.9.2
    # via pandas-profiling
pygments==2.13.0
    # via
    #   ipython
    #   nbconvert
pyopenssl==22.1.0
    # via flytekit
pyparsing==3.0.9
    # via
    #   matplotlib
    #   packaging
pyrsistent==0.18.1
    # via jsonschema
python-dateutil==2.8.2
    # via
    #   arrow
    #   croniter
    #   flytekit
    #   jupyter-client
    #   matplotlib
    #   pandas
python-json-logger==2.0.4
    # via flytekit
python-slugify==6.1.2
    # via cookiecutter
pytimeparse==1.1.8
    # via flytekit
pytz==2022.5
    # via
    #   flytekit
    #   pandas
pywavelets==1.4.1
    # via imagehash
pyyaml==6.0
    # via
    #   cookiecutter
    #   flytekit
    #   pandas-profiling
    #   papermill
pyzmq==24.0.1
    # via
    #   ipykernel
    #   jupyter-client
regex==2022.9.13
    # via docker-image-py
requests==2.28.1
    # via
    #   cookiecutter
    #   docker
    #   flytekit
    #   pandas-profiling
    #   papermill
    #   responses
responses==0.22.0
    # via flytekit
retry==0.9.2
    # via flytekit
scipy==1.9.2
    # via
    #   imagehash
    #   missingno
    #   pandas-profiling
    #   phik
    #   seaborn
    #   statsmodels
seaborn==0.11.2
    # via
    #   missingno
    #   pandas-profiling
six==1.16.0
    # via
    #   asttokens
    #   bleach
    #   grpcio
    #   patsy
    #   python-dateutil
sortedcontainers==2.4.0
    # via flytekit
soupsieve==2.3.2.post1
    # via beautifulsoup4
stack-data==0.5.1
    # via ipython
statsd==3.3.0
    # via flytekit
statsmodels==0.13.2
    # via pandas-profiling
tangled-up-in-unicode==0.2.0
    # via
    #   pandas-profiling
    #   visions
tenacity==8.1.0
    # via
    #   papermill
    #   plotly
text-unidecode==1.3
    # via python-slugify
textwrap3==0.9.2
    # via ansiwrap
tinycss2==1.2.1
    # via nbconvert
toml==0.10.2
    # via responses
tornado==6.2
    # via
    #   ipykernel
    #   jupyter-client
tqdm==4.64.1
    # via
    #   pandas-profiling
    #   papermill
traitlets==5.5.0
    # via
    #   ipykernel
    #   ipython
    #   jupyter-client
    #   jupyter-core
    #   matplotlib-inline
    #   nbclient
    #   nbconvert
    #   nbformat
types-toml==0.10.8
    # via responses
typing-extensions==4.4.0
    # via
    #   flytekit
    #   pydantic
    #   typing-inspect
typing-inspect==0.8.0
    # via dataclasses-json
urllib3==1.26.12
    # via
    #   docker
    #   flytekit
    #   requests
    #   responses
visions[type_image_path]==0.7.5
    # via pandas-profiling
wcwidth==0.2.5
    # via prompt-toolkit
webencodings==0.5.1
    # via
    #   bleach
    #   tinycss2
websocket-client==1.4.1
    # via docker
wheel==0.37.1
    # via
    #   -r ../../../common/requirements-common.in
    #   flytekit
wrapt==1.14.1
    # via
    #   deprecated
    #   flytekit
zipp==3.9.0
    # via importlib-metadata
Also not sure why your requirements has flytekit 0.32.3 but your local system has flytekit 1.2.1
The code works locally but on demo, it’s still throwing the error. @Kevin Su, has this issue not been resolved in 1.2.1 release?
k

Kevin Su

10/19/2022, 11:34 PM
it should work in 1.1.0. there are some breaking changes in 1.2.1. will fix it in 1.2.2
p

Padma Priya M

10/27/2022, 7:21 AM
hi, is this issue resolved in 1.2.2 ? i tried with 1.1.0, facing same issue. not sure if i am doing something wrong. installed all the requirements which u have mentioned by changing only the flytekit version to 1.1.0 and then registered using ghcr.io/flyteorg/flytecookbook:papermilltasks-latest image. the workflow is getting registered successfully but when i create an execution getting this error.
s

Samhita Alla

10/27/2022, 8:43 AM
@Padma Priya M, papermilltasks-latest will have the latest flytekit image. Can you first try running the code locally with flytekit 1.1.0?
p

Padma Priya M

10/27/2022, 9:20 AM
it is working in local with flytekit 1.1.0
s

Samhita Alla

10/27/2022, 9:22 AM
Can you build the image with flytekit 1.1.0 and supply it to the pyflyte run command?
k

Kevin Su

10/27/2022, 9:37 AM
@Padma Priya M sorry, we released early for some reason, so 1.2.2 and 1.2.3 don’t have the fix. I just created a pr, you can give it a try.
Copy code
pip install git+<https://github.com/flyteorg/flytekit@>"papermill-bug-1#egg=flytekitplugins-papermill&subdirectory=plugins/flytekit-papermill"
3 Views
Powered by