<https://github.com/flyteorg/flyte/tree/master|master>

<https://github.com/flyteorg/flyte/commit/b44d0bb84f43308a9710f588a51217beb40a97ae|b44d0bb8>

+ exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=10.0.130.1 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.deploy.PythonRunner local:///opt/venv/bin/entrypoint.py pyflyte-execute --inputs <s3://union-j1-us-east-2-mockjackorg/metadata/propeller/flytesnacks-development-a7wbdflrdzs7sr7lhrff/n0/data/inputs.pb> --output-prefix <s3://union-j1-us-east-2-mockjackorg/metadata/propeller/flytesnacks-development-a7wbdflrdzs7sr7lhrff/n0/data/0> --raw-output-data-prefix <s3://union-j1-us-east-2-mockjackorg/5a/a7wbdflrdzs7sr7lhrff-n0-0> --checkpoint-path <s3://union-j1-us-east-2-mockjackorg/5a/a7wbdflrdzs7sr7lhrff-n0-0/_flytecheckpoints> --prev-checkpoint '""' --resolver flytekit.core.python_auto_container.default_task_resolver -- task-module k8s_spark.pyspark_pi task-name hello_spark
22/12/22 22:55:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Matplotlib created a temporary config/cache directory at /tmp/matplotlib-t41x9d5w because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.

{"asctime": "2022-12-22 22:55:26,788", "name": "flytekit", "levelname": "ERROR", "message": "Error from command '['aws', 's3', 'cp', '<s3://union-j1-us-east-2-mockjackorg/metadata/propeller/flytesnacks-development-a7wbdflrdzs7sr7lhrff/n0/data/inputs.pb>', '/tmp/flyte-bmtatnmx/sandbox/local_flytekit/inputs.pb']':\nb\"fatal error: [Errno 13] Permission denied: '/.aws'\\n\"\n"}
{"asctime": "2022-12-22 22:55:27,312", "name": "flytekit", "levelname": "ERROR", "message": "Error from command '['aws', '--no-sign-request', 's3', 'cp', '<s3://union-j1-us-east-2-mockjackorg/metadata/propeller/flytesnacks-development-a7wbdflrdzs7sr7lhrff/n0/data/inputs.pb>', '/tmp/flyte-bmtatnmx/sandbox/local_flytekit/inputs.pb']':\nb'fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden\\n'\n"}

ClusterCreating

w<random>-n0

w<random>-n1

random

w<random>

pip

protoc-gen-swagger 0.1.0 requires protobuf, which is not installed.
jsonschema 3.2.0 requires pyrsistent, which is not installed.
flyteidl 1.3.1 requires protobuf, which is not installed.
flyteidl 1.3.1 requires googleapis-common-protos, which is not installed.
docker-image-py 0.1.12 requires regex, which is not installed.
binaryornot 0.4.4 requires chardet, which is not installed.

pyflyte run

export FLYTECTL_CONFIG=...

<https://github.com/flyteorg/flytekit/tree/master|master>

<https://github.com/flyteorg/flytekit/commit/425f48812f2d2bf86f2efbdd21f43fcee48d3a1d|425f4881>

pip

protoc-gen-swagger 0.1.0 requires protobuf, which is not installed.
paramiko 2.12.0 requires pynacl, which is not installed.
Keras-Preprocessing 1.1.2 requires numpy, which is not installed.
jsonschema 3.2.0 requires pyrsistent, which is not installed.
google-auth 2.15.0 requires rsa, which is not installed.
flytekit 0.12.10 requires protobuf, which is not installed.
flytekit 0.12.10 requires ipykernel, which is not installed.
flytekit 0.12.10 requires black, which is not installed.
flytekit 0.12.10 has requirement flyteidl<1.0.0,>=0.18.4, but you have flyteidl 1.3.1.
flytekit 0.12.10 has requirement python-dateutil<=2.8.1,>=2.1, but you have python-dateutil 2.8.2.
flytekit 0.12.10 has requirement pytz<2018.5,>=2017.2, but you have pytz 2022.6.
flytekit 0.12.10 has requirement configparser<4.0.0,>=3.0.0, but you have configparser 4.0.2.
flyteidl 1.3.1 requires googleapis-common-protos, which is not installed.
flyteidl 1.3.1 requires protobuf, which is not installed.
docker-image-py 0.1.12 requires regex, which is not installed.

flytectl config init

admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///localhost:30081
  authType: Pkce
  insecure: true
logger:
  show-source: true
  level:

flytectl demo start

admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: localhost:30080
  authType: Pkce
  insecure: true
console:
  endpoint: <http://localhost:30080>
logger:
  show-source: true
  level: 0

flytectl config init

config-sandbox.yaml

Optional[FlyteFile]

from dataclasses import dataclass
from typing import Optional

from dataclasses_json import dataclass_json
from flytekit import workflow, task
from flytekit.types.file import FlyteFile


@dataclass_json
@dataclass
class TaskOutput:
    file1: FlyteFile
    file2: Optional[FlyteFile]

@task
def task1() -> TaskOutput:
    with open("file21.txt", "w+") as f:
        f.write("b")
    with open("file22.txt", "w+") as f:
        f.write("c")

    return TaskOutput(
        FlyteFile("file21.txt", remote_path="<s3://my-s3-bucket/tmp/file21.txt>"),
        FlyteFile("file22.txt", remote_path="<s3://my-s3-bucket/tmp/file22.txt>")
    )

@task
def assert_values_in_dataclass(to: TaskOutput):
    assert to.file1.remote_source == "<s3://my-s3-bucket/tmp/file22.txt>"
    assert to.file2.remote_source == "<s3://my-s3-bucket/tmp/file22.txt>"


@workflow
def wf():
    to = task1()
    assert_values_in_dataclass(to=to)

Optional[FlyteFile]

FastSerializationSettings

pip

protoc-gen-swagger 0.1.0 requires protobuf, which is not installed.
jsonschema 3.2.0 requires pyrsistent, which is not installed.
flyteidl 1.3.1 requires protobuf, which is not installed.
flyteidl 1.3.1 requires googleapis-common-protos, which is not installed.
docker-image-py 0.1.12 requires regex, which is not installed.
binaryornot 0.4.4 requires chardet, which is not installed.

pip

protoc-gen-swagger 0.1.0 requires protobuf, which is not installed.
google-auth 2.15.0 requires rsa, which is not installed.
flyteidl 1.3.0 requires protobuf, which is not installed.
flyteidl 1.3.0 requires googleapis-common-protos, which is not installed.
docker-image-py 0.1.12 requires regex, which is not installed.

pip

vaex 4.16.0 requires vaex-core, which is not installed.
vaex-viz 0.5.4 requires vaex-core, which is not installed.
vaex-server 0.8.1 requires vaex-core, which is not installed.
vaex-server 0.8.1 requires fastapi, which is not installed.
vaex-server 0.8.1 requires uvicorn, which is not installed.
vaex-ml 0.18.1 requires vaex-core, which is not installed.
vaex-jupyter 0.8.1 requires vaex-core, which is not installed.
vaex-jupyter 0.8.1 requires xarray, which is not installed.
vaex-hdf5 0.14.1 requires vaex-core, which is not installed.
vaex-astro 0.9.3 requires vaex-core, which is not installed.
protoc-gen-swagger 0.1.0 requires protobuf, which is not installed.
notebook 5.7.16 requires terminado, which is not installed.
nbformat 4.4.0 requires jsonschema, which is not installed.
nbconvert 5.6.1 has requirement mistune<2,>=0.8.1, but you have mistune 2.0.4.
jupyter 1.0.0 requires qtconsole, which is not installed.
jupyter 1.0.0 requires jupyter-console, which is not installed.
ipyvolume 0.5.2 requires pythreejs, which is not installed.
ipython 5.10.0 requires simplegeneric, which is not installed.
ipympl 0.9.2 has requirement matplotlib<4,>=3.4.0, but you have matplotlib 2.2.5.
ipyleaflet 0.17.0 requires xyzservices, which is not installed.
ImageHash 4.3.1 requires scipy, which is not installed.
ImageHash 4.3.1 requires PyWavelets, which is not installed.
google-auth 2.15.0 requires rsa, which is not installed.
flyteidl 1.3.1 requires googleapis-common-protos, which is not installed.
flyteidl 1.3.1 requires protobuf, which is not installed.
docker-image-py 0.1.12 requires regex, which is not installed.

pip

sagemaker-training 3.9.2 requires protobuf, which is not installed.
sagemaker-training 3.9.2 requires werkzeug, which is not installed.
sagemaker-training 3.9.2 requires scipy, which is not installed.
protoc-gen-swagger 0.1.0 requires protobuf, which is not installed.
paramiko 2.12.0 requires pynacl, which is not installed.
flyteidl 1.3.0 requires protobuf, which is not installed.
flyteidl 1.3.0 requires googleapis-common-protos, which is not installed.
docker-image-py 0.1.12 requires regex, which is not installed.
binaryornot 0.4.4 requires chardet, which is not installed.

pip

protoc-gen-swagger 0.1.0 requires protobuf, which is not installed.
flyteidl 1.3.0 requires protobuf, which is not installed.
flyteidl 1.3.0 requires googleapis-common-protos, which is not installed.
docker-image-py 0.1.12 requires regex, which is not installed.
binaryornot 0.4.4 requires chardet, which is not installed.

pip

protoc-gen-swagger 0.1.0 requires protobuf, which is not installed.
google-auth 2.15.0 requires rsa, which is not installed.
flyteidl 1.3.0 requires protobuf, which is not installed.
flyteidl 1.3.0 requires googleapis-common-protos, which is not installed.
docker-image-py 0.1.12 requires regex, which is not installed.

Optional

FlyteFile

StructuredDataset

No such file or directory: /var/folders/wq/3hjh3ms916b6dj56zx0f_x000000gq/T/flyte-69d2tww2/sandbox/local_flytekit/95bac8efeb64a8d10d34c73b66df7051/00000

00000

StructuredDataset

<https://github.com/flyteorg/flytekit/tree/master|master>

<https://github.com/flyteorg/flytekit/commit/242a11cf1fe564d6310eda4f1df4f8b7cb8f334a|242a11cf>

{"asctime": "2022-12-25 22:57:07,317", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.PandasToParquetEncodingHandler object at 0x7fce624a8220> with format parquet as default for <class 'pandas.core.frame.DataFrame'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,317", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.PandasToParquetEncodingHandler object at 0x7fce624a8220> with format parquet as default for <class 'pandas.core.frame.DataFrame'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,317", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.PandasToParquetEncodingHandler object at 0x7fce624a8220> with format parquet as default for <class 'pandas.core.frame.DataFrame'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,317", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.PandasToParquetEncodingHandler object at 0x7fce624a8220> with format parquet as default for <class 'pandas.core.frame.DataFrame'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,317", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ParquetToPandasDecodingHandler object at 0x7fce624549a0> with format parquet as default for <class 'pandas.core.frame.DataFrame'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,317", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ParquetToPandasDecodingHandler object at 0x7fce624549a0> with format parquet as default for <class 'pandas.core.frame.DataFrame'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,317", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ParquetToPandasDecodingHandler object at 0x7fce624549a0> with format parquet as default for <class 'pandas.core.frame.DataFrame'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,317", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ParquetToPandasDecodingHandler object at 0x7fce624549a0> with format parquet as default for <class 'pandas.core.frame.DataFrame'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,317", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ParquetToPandasDecodingHandler object at 0x7fce624549a0> with format parquet as default for <class 'pandas.core.frame.DataFrame'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,317", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ArrowToParquetEncodingHandler object at 0x7fce62454940> with format parquet as default for <class 'pyarrow.lib.Table'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,318", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ArrowToParquetEncodingHandler object at 0x7fce62454940> with format parquet as default for <class 'pyarrow.lib.Table'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,318", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ArrowToParquetEncodingHandler object at 0x7fce62454940> with format parquet as default for <class 'pyarrow.lib.Table'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,318", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ArrowToParquetEncodingHandler object at 0x7fce62454940> with format parquet as default for <class 'pyarrow.lib.Table'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,318", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ParquetToArrowDecodingHandler object at 0x7fce624a82b0> with format parquet as default for <class 'pyarrow.lib.Table'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,318", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ParquetToArrowDecodingHandler object at 0x7fce624a82b0> with format parquet as default for <class 'pyarrow.lib.Table'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,318", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ParquetToArrowDecodingHandler object at 0x7fce624a82b0> with format parquet as default for <class 'pyarrow.lib.Table'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,318", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ParquetToArrowDecodingHandler object at 0x7fce624a82b0> with format parquet as default for <class 'pyarrow.lib.Table'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,318", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ParquetToArrowDecodingHandler object at 0x7fce624a82b0> with format parquet as default for <class 'pyarrow.lib.Table'>, parquet already specified."}

ipywidgets

[0]: code:"UnknownError" message:"Pod failed. No message received from kubernetes.\r\n[f773f1ce96ea54485bd2-n0-0-0] terminated with exit code (1). Reason [Error]. Message: \nandler <flytekit.types.structured.basic_dfs.ParquetToArrowDecodingHandler object at 0x7f29ae5f4fa0> with format parquet as default for <class 'pyarrow.lib.Table'>, parquet already specified.\"}\n{\"asctime\": \"2022-12-26 06:20:50,244\", \"name\": \"flytekit\", \"levelname\": \"WARNING\", \"message\": \"Not using handler <flytekit.types.structured.basic_dfs.ParquetToArrowDecodingHandler object at 0x7f29ae5f4fa0> with format parquet as default for <class 'pyarrow.lib.Table'>, parquet already specified.\"}\nTraceback (most recent call last):\n  File \"/usr/local/bin/pyflyte-fast-execute\", line 5, in <module>\n    from flytekit.bin.entrypoint import fast_execute_task_cmd\n  File \"/usr/local/lib/python3.9/site-packages/flytekit/__init__.py\", line 261, in <module>\n    load_implicit_plugins()\n  File \"/usr/local/lib/python3.9/site-packages/flytekit/__init__.py\", line 257, in load_implicit_plugins\n    p.load()\n  File \"/usr/local/lib/python3.9/site-packages/importlib_metadata/__init__.py\", line 208, in load\n    module = import_module(match.group('module'))\n  File \"/usr/local/lib/python3.9/importlib/__init__.py\", line 127, in import_module\n    return _bootstrap._gcd_import(name[level:], package, level)\n  File \"/usr/local/lib/python3.9/site-packages/flytekitplugins/deck/__init__.py\", line 15, in <module>\n    from .renderer import BoxRenderer, FrameProfilingRenderer, MarkdownRenderer\n  File \"/usr/local/lib/python3.9/site-packages/flytekitplugins/deck/renderer.py\", line 4, in <module>\n    from pandas_profiling import ProfileReport\n  File \"/usr/local/lib/python3.9/site-packages/pandas_profiling/__init__.py\", line 6, in <module>\n    from pandas_profiling.compare_reports import compare\n  File \"/usr/local/lib/python3.9/site-packages/pandas_profiling/compare_reports.py\", line 7, in <module>\n    from pandas_profiling.profile_report import ProfileReport\n  File \"/usr/local/lib/python3.9/site-packages/pandas_profiling/profile_report.py\", line 9, in <module>\n    from ipywidgets.widgets.widget_box import VBox\nModuleNotFoundError: No module named 'ipywidgets'\n."