GitHub
12/22/2022, 11:05 PMGitHub
12/22/2022, 11:12 PMGitHub
12/22/2022, 11:34 PM<https://github.com/flyteorg/flyte/tree/master|master>
by jeevb
<https://github.com/flyteorg/flyte/commit/b44d0bb84f43308a9710f588a51217beb40a97ae|b44d0bb8>
- Updates to flyte-binary chart to support plugins (#3183)
flyteorg/flyteGitHub
12/22/2022, 11:41 PM+ exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=10.0.130.1 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.deploy.PythonRunner local:///opt/venv/bin/entrypoint.py pyflyte-execute --inputs <s3://union-j1-us-east-2-mockjackorg/metadata/propeller/flytesnacks-development-a7wbdflrdzs7sr7lhrff/n0/data/inputs.pb> --output-prefix <s3://union-j1-us-east-2-mockjackorg/metadata/propeller/flytesnacks-development-a7wbdflrdzs7sr7lhrff/n0/data/0> --raw-output-data-prefix <s3://union-j1-us-east-2-mockjackorg/5a/a7wbdflrdzs7sr7lhrff-n0-0> --checkpoint-path <s3://union-j1-us-east-2-mockjackorg/5a/a7wbdflrdzs7sr7lhrff-n0-0/_flytecheckpoints> --prev-checkpoint '""' --resolver flytekit.core.python_auto_container.default_task_resolver -- task-module k8s_spark.pyspark_pi task-name hello_spark
22/12/22 22:55:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Matplotlib created a temporary config/cache directory at /tmp/matplotlib-t41x9d5w because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
{"asctime": "2022-12-22 22:55:26,788", "name": "flytekit", "levelname": "ERROR", "message": "Error from command '['aws', 's3', 'cp', '<s3://union-j1-us-east-2-mockjackorg/metadata/propeller/flytesnacks-development-a7wbdflrdzs7sr7lhrff/n0/data/inputs.pb>', '/tmp/flyte-bmtatnmx/sandbox/local_flytekit/inputs.pb']':\nb\"fatal error: [Errno 13] Permission denied: '/.aws'\\n\"\n"}
{"asctime": "2022-12-22 22:55:27,312", "name": "flytekit", "levelname": "ERROR", "message": "Error from command '['aws', '--no-sign-request', 's3', 'cp', '<s3://union-j1-us-east-2-mockjackorg/metadata/propeller/flytesnacks-development-a7wbdflrdzs7sr7lhrff/n0/data/inputs.pb>', '/tmp/flyte-bmtatnmx/sandbox/local_flytekit/inputs.pb']':\nb'fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden\\n'\n"}
Removing the usage of spark-user-id
flyteorg/flytesnacks
✅ All checks have passed
1/1 successful checksGitHub
12/23/2022, 7:36 AMClusterCreating
state for tasks would be helpful for people to know what's going on underneath. It is also much easier to debug if an operator fails to create a new cluster. In addition, cluster creation time and current task running time can be measured.it
Goal: What should the final outcome look like, ideally?
Users will know Flyte is creating a cluster when the node is stared
Describe alternatives you've considered
NA
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
12/23/2022, 10:13 AMw<random>-n0
, w<random>-n1
, etc. where random
is a random integer generated per workflow build, and w<random>
can be specified by a user; and to make a node readable, node name is inferred from class name.
Tracking Issue
Follow-up issue
NA
flyteorg/flytekit-java
✅ All checks have passed
3/3 successful checksGitHub
12/23/2022, 10:15 AMGitHub
12/23/2022, 5:09 PMpip
dependencies of this project.
Changes included in this PR
• Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
• requirements.txt
⚠️ Warning
protoc-gen-swagger 0.1.0 requires protobuf, which is not installed.
jsonschema 3.2.0 requires pyrsistent, which is not installed.
flyteidl 1.3.1 requires protobuf, which is not installed.
flyteidl 1.3.1 requires googleapis-common-protos, which is not installed.
docker-image-py 0.1.12 requires regex, which is not installed.
binaryornot 0.4.4 requires chardet, which is not installed.
Vulnerabilities that will be fixed
By pinning:
(*) Note that the real score may have changed since the PR was raised.
Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.
Check the changes in this PR to ensure they won't cause issues with your project.
* * *
Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information: https://camo.githubusercontent.com/eed6f108d5c917b9f136849dcc551d9621ee15c1356ee11c6dcf1f75b826136e/68747470733a2f2f6170692e7365676d656e742e696f2f76312f706978656c2f747261636b3f646174613d65794a33636d6c305a55746c65534936496e4a79576d785a634564485932527954485a7362306c596430645563566734576b4652546e4e434f5545774969776959573576626e6c746233567a535751694f694a6d4f544d354f4745334d793034595755324c54526d596d4d744f546c6d5969303359544e6b4d6a426a595459334e5449694c434a6c646d567564434936496c425349485a705a58646c5a434973496e42796233426c636e52705a584d694f6e736963484a4a5a434936496d59354d7a6b345954637a4c5468685a5459744e475a69597930354f575a694c5464684d3251794d474e684e6a63314d694a3966513d3d
🧐 View latest project report
🛠️ Adjust project settings
📚 Read more about Snyk's upgrade and patch logic
* * *
Learn how to fix vulnerabilities with free interactive lessons:
🦉 Regular Expression Denial of Service (ReDoS)
flyteorg/flytekitGitHub
12/23/2022, 5:13 PMpyflyte run
against the flytectl demo cluster. This change means that they no longer have to export FLYTECTL_CONFIG=...
to get it working.
Signed-off-by: Niels Bantilan niels.bantilan@gmail.com
Type
☑︎ Bug Fix
☐ Feature
☐ Plugin
Are all requirements met?
☐ Code completed
☐ Smoke tested
☐ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
How did you fix the bug, make the feature etc. Link to any design docs etc
Tracking Issue
https://github.com/flyteorg/flyte/issues/
Follow-up issue
NA
OR
https://github.com/flyteorg/flyte/issues/
flyteorg/flytekitGitHub
12/23/2022, 5:35 PMpip
dependencies of this project.
Changes included in this PR
• Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
• doc-requirements.txt
Vulnerabilities that will be fixed
By pinning:
(*) Note that the real score may have changed since the PR was raised.
Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.
Check the changes in this PR to ensure they won't cause issues with your project.
* * *
Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information: https://camo.githubusercontent.com/3d4eee94e619a1e296ec74157acc3ecd029a5e68a6fa1dadb9e68870f32f75f3/68747470733a2f2f6170692e7365676d656e742e696f2f76312f706978656c2f747261636b3f646174613d65794a33636d6c305a55746c65534936496e4a79576d785a634564485932527954485a7362306c596430645563566734576b4652546e4e434f5545774969776959573576626e6c746233567a535751694f694a695a444d305957466c4d69316c5954677a4c5451304e6a6b745954686a5a43316c4d546b304e324a6b4e5751784d5441694c434a6c646d567564434936496c425349485a705a58646c5a434973496e42796233426c636e52705a584d694f6e736963484a4a5a434936496d4a6b4d7a5268595755794c5756684f444d744e4451324f5331684f474e6b4c5755784f545133596d51315a4445784d434a3966513d3d
🧐 View latest project report
🛠️ Adjust project settings
📚 Read more about Snyk's upgrade and patch logic
* * *
Learn how to fix vulnerabilities with free interactive lessons:
🦉 Regular Expression Denial of Service (ReDoS)
flyteorg/flytectlGitHub
12/23/2022, 6:41 PMpip
dependencies of this project.
Changes included in this PR
• Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
• doc-requirements.txt
Vulnerabilities that will be fixed
By pinning:
(*) Note that the real score may have changed since the PR was raised.
Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.
Check the changes in this PR to ensure they won't cause issues with your project.
* * *
Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information: https://camo.githubusercontent.com/5618f5399cc82bd6fbc1ff9c62f8c53210d8986a907d9f51f4a29eaeb2fc51bb/68747470733a2f2f6170692e7365676d656e742e696f2f76312f706978656c2f747261636b3f646174613d65794a33636d6c305a55746c65534936496e4a79576d785a634564485932527954485a7362306c596430645563566734576b4652546e4e434f5545774969776959573576626e6c746233567a535751694f6949305a6d457a596d526c4e5330775a544e6a4c5452685a6a4974596a466b4d4330784d7a4a6a4e7a51775a546c6c4d4459694c434a6c646d567564434936496c425349485a705a58646c5a434973496e42796233426c636e52705a584d694f6e736963484a4a5a434936496a526d59544e695a4755314c54426c4d324d744e47466d4d6931694d5751774c54457a4d6d4d334e44426c4f5755774e694a3966513d3d
🧐 View latest project report
🛠️ Adjust project settings
📚 Read more about Snyk's upgrade and patch logic
* * *
Learn how to fix vulnerabilities with free interactive lessons:
🦉 Regular Expression Denial of Service (ReDoS)
flyteorg/flyteidlGitHub
12/23/2022, 7:14 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytekit/commit/425f48812f2d2bf86f2efbdd21f43fcee48d3a1d|425f4881>
- fix: dev-requirements.txt to reduce vulnerabilities (#1374)
flyteorg/flytekitGitHub
12/23/2022, 7:18 PMpip
dependencies of this project.
Changes included in this PR
• Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
• dev-requirements.txt
⚠️ Warning
protoc-gen-swagger 0.1.0 requires protobuf, which is not installed.
paramiko 2.12.0 requires pynacl, which is not installed.
Keras-Preprocessing 1.1.2 requires numpy, which is not installed.
jsonschema 3.2.0 requires pyrsistent, which is not installed.
google-auth 2.15.0 requires rsa, which is not installed.
flytekit 0.12.10 requires protobuf, which is not installed.
flytekit 0.12.10 requires ipykernel, which is not installed.
flytekit 0.12.10 requires black, which is not installed.
flytekit 0.12.10 has requirement flyteidl<1.0.0,>=0.18.4, but you have flyteidl 1.3.1.
flytekit 0.12.10 has requirement python-dateutil<=2.8.1,>=2.1, but you have python-dateutil 2.8.2.
flytekit 0.12.10 has requirement pytz<2018.5,>=2017.2, but you have pytz 2022.6.
flytekit 0.12.10 has requirement configparser<4.0.0,>=3.0.0, but you have configparser 4.0.2.
flyteidl 1.3.1 requires googleapis-common-protos, which is not installed.
flyteidl 1.3.1 requires protobuf, which is not installed.
docker-image-py 0.1.12 requires regex, which is not installed.
Vulnerabilities that will be fixed
By pinning:
Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.
Check the changes in this PR to ensure they won't cause issues with your project.
* * *
Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information: https://camo.githubusercontent.com/4ccad22e666ea54f02278ae5ea3a0c7a59e30112e66c7d47dc718ca5aca027ba/68747470733a2f2f6170692e7365676d656e742e696f2f76312f706978656c2f747261636b3f646174613d65794a33636d6c305a55746c65534936496e4a79576d785a634564485932527954485a7362306c596430645563566734576b4652546e4e434f5545774969776959573576626e6c746233567a535751694f6949335a6a4d7a5a444979597930344d446b334c5451355a4759744f4463314e43316c4e4459304d574a684d7a526a5a4755694c434a6c646d567564434936496c425349485a705a58646c5a434973496e42796233426c636e52705a584d694f6e736963484a4a5a434936496a646d4d7a4e6b4d6a4a6a4c5467774f5463744e446c6b5a6930344e7a55304c5755304e6a5178596d457a4e474e6b5a534a3966513d3d
🧐 View latest project report
🛠️ Adjust project settings
📚 Read more about Snyk's upgrade and patch logic
* * *
Learn how to fix vulnerabilities with free interactive lessons:
🦉 Regular Expression Denial of Service (ReDoS)
flyteorg/flytekitGitHub
12/23/2022, 8:55 PMflytectl config init
is:
admin:
# For GRPC endpoints you might want to use dns:///flyte.myexample.com
endpoint: dns:///localhost:30081
authType: Pkce
insecure: true
logger:
show-source: true
level:
But the config produced by flytectl demo start
is:
admin:
# For GRPC endpoints you might want to use dns:///flyte.myexample.com
endpoint: localhost:30080
authType: Pkce
insecure: true
console:
endpoint: <http://localhost:30080>
logger:
show-source: true
level: 0
This may be confusing for new users who would expect the two configs to be consistent
Provide a possible output or UX example
The output of of flytectl config init
should be the same as config-sandbox.yaml
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
12/23/2022, 9:23 PMGitHub
12/23/2022, 9:42 PMOptional[FlyteFile]
correctly.
For example, this workflow fails:
from dataclasses import dataclass
from typing import Optional
from dataclasses_json import dataclass_json
from flytekit import workflow, task
from flytekit.types.file import FlyteFile
@dataclass_json
@dataclass
class TaskOutput:
file1: FlyteFile
file2: Optional[FlyteFile]
@task
def task1() -> TaskOutput:
with open("file21.txt", "w+") as f:
f.write("b")
with open("file22.txt", "w+") as f:
f.write("c")
return TaskOutput(
FlyteFile("file21.txt", remote_path="<s3://my-s3-bucket/tmp/file21.txt>"),
FlyteFile("file22.txt", remote_path="<s3://my-s3-bucket/tmp/file22.txt>")
)
@task
def assert_values_in_dataclass(to: TaskOutput):
assert to.file1.remote_source == "<s3://my-s3-bucket/tmp/file22.txt>"
assert to.file2.remote_source == "<s3://my-s3-bucket/tmp/file22.txt>"
@workflow
def wf():
to = task1()
assert_values_in_dataclass(to=to)
Expected behavior
Optional[FlyteFile]
should be allowed in Dataclasses.
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
12/23/2022, 10:35 PMpip
dependencies of this project.
Changes included in this PR
• Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
• doc-requirements.txt
Vulnerabilities that will be fixed
By pinning:
(*) Note that the real score may have changed since the PR was raised.
Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.
Check the changes in this PR to ensure they won't cause issues with your project.
* * *
Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information: https://camo.githubusercontent.com/57b4314e806840f6f14ecc2e77c9abb9648c62129cf97750bc2763c3fbaa7339/68747470733a2f2f6170692e7365676d656e742e696f2f76312f706978656c2f747261636b3f646174613d65794a33636d6c305a55746c65534936496e4a79576d785a634564485932527954485a7362306c596430645563566734576b4652546e4e434f5545774969776959573576626e6c746233567a535751694f6949314e446b774e444a6b4d43316c4e4745354c5451334f4449744f47466c5a4331685a5463304e7a6c6c5a5445774d4441694c434a6c646d567564434936496c425349485a705a58646c5a434973496e42796233426c636e52705a584d694f6e736963484a4a5a434936496a55304f5441304d6d51774c57553059546b744e4463344d6930345957566b4c57466c4e7a51334f57566c4d5441774d434a3966513d3d
🧐 View latest project report
🛠️ Adjust project settings
📚 Read more about Snyk's upgrade and patch logic
* * *
Learn how to fix vulnerabilities with free interactive lessons:
🦉 Regular Expression Denial of Service (ReDoS)
flyteorg/flyteGitHub
12/23/2022, 10:53 PMFastSerializationSettings
to the API reference documentation.
Type
☑︎ Doc
☐ Bug Fix
☐ Feature
☐ Plugin
Are all requirements met?
☐ Code completed
☐ Smoke tested
☐ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
How did you fix the bug, make the feature etc. Link to any design docs etc
Tracking Issue
https://github.com/flyteorg/flyte/issues/
Follow-up issue
NA
OR
https://github.com/flyteorg/flyte/issues/
flyteorg/flytekitGitHub
12/23/2022, 10:58 PMpip
dependencies of this project.
Changes included in this PR
• Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
• requirements-spark2.txt
⚠️ Warning
protoc-gen-swagger 0.1.0 requires protobuf, which is not installed.
jsonschema 3.2.0 requires pyrsistent, which is not installed.
flyteidl 1.3.1 requires protobuf, which is not installed.
flyteidl 1.3.1 requires googleapis-common-protos, which is not installed.
docker-image-py 0.1.12 requires regex, which is not installed.
binaryornot 0.4.4 requires chardet, which is not installed.
Vulnerabilities that will be fixed
By pinning:
(*) Note that the real score may have changed since the PR was raised.
Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.
Check the changes in this PR to ensure they won't cause issues with your project.
* * *
Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information: https://camo.githubusercontent.com/869f31d4dcd7fd43622e3dcb1387dca7c3c881f31ff6005f935594bc16f5cafb/68747470733a2f2f6170692e7365676d656e742e696f2f76312f706978656c2f747261636b3f646174613d65794a33636d6c305a55746c65534936496e4a79576d785a634564485932527954485a7362306c596430645563566734576b4652546e4e434f5545774969776959573576626e6c746233567a535751694f6949355a5467774e446c6c4d6930784d574d314c5451314d574d7459545535597930344f544a6c4e6a4d334f474e6a4d6a4d694c434a6c646d567564434936496c425349485a705a58646c5a434973496e42796233426c636e52705a584d694f6e736963484a4a5a434936496a6c6c4f4441304f5755794c544578597a55744e445578597931684e546c6a4c5467354d6d55324d7a633459324d794d794a3966513d3d
🧐 View latest project report
🛠️ Adjust project settings
📚 Read more about Snyk's upgrade and patch logic
* * *
Learn how to fix vulnerabilities with free interactive lessons:
🦉 Regular Expression Denial of Service (ReDoS)
flyteorg/flytekitGitHub
12/23/2022, 11:19 PMpip
dependencies of this project.
Changes included in this PR
• Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
• plugins/flytekit-papermill/dev-requirements.txt
⚠️ Warning
protoc-gen-swagger 0.1.0 requires protobuf, which is not installed.
google-auth 2.15.0 requires rsa, which is not installed.
flyteidl 1.3.0 requires protobuf, which is not installed.
flyteidl 1.3.0 requires googleapis-common-protos, which is not installed.
docker-image-py 0.1.12 requires regex, which is not installed.
Vulnerabilities that will be fixed
By pinning:
(*) Note that the real score may have changed since the PR was raised.
Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.
Check the changes in this PR to ensure they won't cause issues with your project.
* * *
Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information: https://camo.githubusercontent.com/2eebced9524062cabcfab0399b6fb06e44501363fcd1f6feb9f5c54ccccdad07/68747470733a2f2f6170692e7365676d656e742e696f2f76312f706978656c2f747261636b3f646174613d65794a33636d6c305a55746c65534936496e4a79576d785a634564485932527954485a7362306c596430645563566734576b4652546e4e434f5545774969776959573576626e6c746233567a535751694f6949344d3251304e5745344d4330314f54466d4c5451325a444174595759784d4330325a4752684e444d324d6a6b79596d49694c434a6c646d567564434936496c425349485a705a58646c5a434973496e42796233426c636e52705a584d694f6e736963484a4a5a434936496a677a5a445131595467774c5455354d5759744e445a6b4d4331685a6a45774c545a6b5a4745304d7a59794f544a6959694a3966513d3d
🧐 View latest project report
🛠️ Adjust project settings
📚 Read more about Snyk's upgrade and patch logic
* * *
Learn how to fix vulnerabilities with free interactive lessons:
🦉 Regular Expression Denial of Service (ReDoS)
flyteorg/flytekitGitHub
12/24/2022, 12:25 AMpip
dependencies of this project.
Changes included in this PR
• Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
• doc-requirements.txt
⚠️ Warning
vaex 4.16.0 requires vaex-core, which is not installed.
vaex-viz 0.5.4 requires vaex-core, which is not installed.
vaex-server 0.8.1 requires vaex-core, which is not installed.
vaex-server 0.8.1 requires fastapi, which is not installed.
vaex-server 0.8.1 requires uvicorn, which is not installed.
vaex-ml 0.18.1 requires vaex-core, which is not installed.
vaex-jupyter 0.8.1 requires vaex-core, which is not installed.
vaex-jupyter 0.8.1 requires xarray, which is not installed.
vaex-hdf5 0.14.1 requires vaex-core, which is not installed.
vaex-astro 0.9.3 requires vaex-core, which is not installed.
protoc-gen-swagger 0.1.0 requires protobuf, which is not installed.
notebook 5.7.16 requires terminado, which is not installed.
nbformat 4.4.0 requires jsonschema, which is not installed.
nbconvert 5.6.1 has requirement mistune<2,>=0.8.1, but you have mistune 2.0.4.
jupyter 1.0.0 requires qtconsole, which is not installed.
jupyter 1.0.0 requires jupyter-console, which is not installed.
ipyvolume 0.5.2 requires pythreejs, which is not installed.
ipython 5.10.0 requires simplegeneric, which is not installed.
ipympl 0.9.2 has requirement matplotlib<4,>=3.4.0, but you have matplotlib 2.2.5.
ipyleaflet 0.17.0 requires xyzservices, which is not installed.
ImageHash 4.3.1 requires scipy, which is not installed.
ImageHash 4.3.1 requires PyWavelets, which is not installed.
google-auth 2.15.0 requires rsa, which is not installed.
flyteidl 1.3.1 requires googleapis-common-protos, which is not installed.
flyteidl 1.3.1 requires protobuf, which is not installed.
docker-image-py 0.1.12 requires regex, which is not installed.
Vulnerabilities that will be fixed
By pinning:
(*) Note that the real score may have changed since the PR was raised.
Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.
Check the changes in this PR to ensure they won't cause issues with your project.
* * *
Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information: https://camo.githubusercontent.com/1a3952c1b33b19e72d1f99106e19d20f6b5e7691761500ce3f2af6079abb4d47/68747470733a2f2f6170692e7365676d656e742e696f2f76312f706978656c2f747261636b3f646174613d65794a33636d6c305a55746c65534936496e4a79576d785a634564485932527954485a7362306c596430645563566734576b4652546e4e434f5545774969776959573576626e6c746233567a535751694f69497a4d3245314e574e684f4330784e6d55344c5451784d6a4574596a49314d53316d5a6a646c5a6d517a5a57566c596a67694c434a6c646d567564434936496c425349485a705a58646c5a434973496e42796233426c636e52705a584d694f6e736963484a4a5a434936496a4d7a59545531593245344c5445325a5467744e4445794d5331694d6a55784c575a6d4e32566d5a444e6c5a5756694f434a3966513d3d
🧐 View latest project report
🛠️ Adjust project settings
📚 Read more about Snyk's upgrade and patch logic
* * *
Learn how to fix vulnerabilities with free interactive lessons:
🦉 Regular Expression Denial of Service (ReDoS)
flyteorg/flytekitGitHub
12/24/2022, 4:45 AMpip
dependencies of this project.
Changes included in this PR
• Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
• plugins/flytekit-aws-sagemaker/requirements.txt
⚠️ Warning
sagemaker-training 3.9.2 requires protobuf, which is not installed.
sagemaker-training 3.9.2 requires werkzeug, which is not installed.
sagemaker-training 3.9.2 requires scipy, which is not installed.
protoc-gen-swagger 0.1.0 requires protobuf, which is not installed.
paramiko 2.12.0 requires pynacl, which is not installed.
flyteidl 1.3.0 requires protobuf, which is not installed.
flyteidl 1.3.0 requires googleapis-common-protos, which is not installed.
docker-image-py 0.1.12 requires regex, which is not installed.
binaryornot 0.4.4 requires chardet, which is not installed.
Vulnerabilities that will be fixed
By pinning:
(*) Note that the real score may have changed since the PR was raised.
Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.
Check the changes in this PR to ensure they won't cause issues with your project.
* * *
Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information: https://camo.githubusercontent.com/2e949105155a7886670cf632dfa6b205648b69de16071b76994b93cc3d3dc24e/68747470733a2f2f6170692e7365676d656e742e696f2f76312f706978656c2f747261636b3f646174613d65794a33636d6c305a55746c65534936496e4a79576d785a634564485932527954485a7362306c596430645563566734576b4652546e4e434f5545774969776959573576626e6c746233567a535751694f694a6d4f4751314e7a59355a4330344e4463354c54526c4e324d74596d59325a4330795a6a6b795957526b5932566c4e4441694c434a6c646d567564434936496c425349485a705a58646c5a434973496e42796233426c636e52705a584d694f6e736963484a4a5a434936496d59345a4455334e6a6c6b4c5467304e7a6b744e475533597931695a6a5a6b4c544a6d4f544a685a47526a5a5755304d434a3966513d3d
🧐 View latest project report
🛠️ Adjust project settings
📚 Read more about Snyk's upgrade and patch logic
* * *
Learn how to fix vulnerabilities with free interactive lessons:
🦉 Regular Expression Denial of Service (ReDoS)
flyteorg/flytekitGitHub
12/24/2022, 4:54 AMpip
dependencies of this project.
Changes included in this PR
• Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
• plugins/flytekit-greatexpectations/requirements.txt
⚠️ Warning
protoc-gen-swagger 0.1.0 requires protobuf, which is not installed.
flyteidl 1.3.0 requires protobuf, which is not installed.
flyteidl 1.3.0 requires googleapis-common-protos, which is not installed.
docker-image-py 0.1.12 requires regex, which is not installed.
binaryornot 0.4.4 requires chardet, which is not installed.
Vulnerabilities that will be fixed
By pinning:
(*) Note that the real score may have changed since the PR was raised.
Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.
Check the changes in this PR to ensure they won't cause issues with your project.
* * *
Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information: https://camo.githubusercontent.com/8e356b875947d6d22ee1880bd676e9bf809ae16296b628aa57f1178202f16f0c/68747470733a2f2f6170692e7365676d656e742e696f2f76312f706978656c2f747261636b3f646174613d65794a33636d6c305a55746c65534936496e4a79576d785a634564485932527954485a7362306c596430645563566734576b4652546e4e434f5545774969776959573576626e6c746233567a535751694f6949784f5751784d6a59334d43307a4d5468694c5451315a474d74596a4d354e793169596a557a59546c6c4d6a646b4f5745694c434a6c646d567564434936496c425349485a705a58646c5a434973496e42796233426c636e52705a584d694f6e736963484a4a5a434936496a45355a4445794e6a63774c544d784f4749744e44566b597931694d7a6b334c574a694e544e684f5755794e32513559534a3966513d3d
🧐 View latest project report
🛠️ Adjust project settings
📚 Read more about Snyk's upgrade and patch logic
* * *
Learn how to fix vulnerabilities with free interactive lessons:
🦉 Regular Expression Denial of Service (ReDoS)
flyteorg/flytekitGitHub
12/24/2022, 5:00 AMpip
dependencies of this project.
Changes included in this PR
• Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
• plugins/flytekit-k8s-pod/requirements.txt
⚠️ Warning
protoc-gen-swagger 0.1.0 requires protobuf, which is not installed.
google-auth 2.15.0 requires rsa, which is not installed.
flyteidl 1.3.0 requires protobuf, which is not installed.
flyteidl 1.3.0 requires googleapis-common-protos, which is not installed.
docker-image-py 0.1.12 requires regex, which is not installed.
Vulnerabilities that will be fixed
By pinning:
(*) Note that the real score may have changed since the PR was raised.
Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.
Check the changes in this PR to ensure they won't cause issues with your project.
* * *
Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information: https://camo.githubusercontent.com/1fa29fa4901fc7f245316a2d68ebc10e3cdcf28b7abda1eec967aa0b6d480306/68747470733a2f2f6170692e7365676d656e742e696f2f76312f706978656c2f747261636b3f646174613d65794a33636d6c305a55746c65534936496e4a79576d785a634564485932527954485a7362306c596430645563566734576b4652546e4e434f5545774969776959573576626e6c746233567a535751694f6949794e444934595759784e5330784d4467324c5451314d5441745954646a4d793035593251324e5745334e6a5a694e4455694c434a6c646d567564434936496c425349485a705a58646c5a434973496e42796233426c636e52705a584d694f6e736963484a4a5a434936496a49304d6a68685a6a45314c5445774f4459744e4455784d4331684e324d7a4c546c6a5a445931595463324e6d49304e534a3966513d3d
🧐 View latest project report
🛠️ Adjust project settings
📚 Read more about Snyk's upgrade and patch logic
* * *
Learn how to fix vulnerabilities with free interactive lessons:
🦉 Regular Expression Denial of Service (ReDoS)
flyteorg/flytekitGitHub
12/24/2022, 1:36 PMOptional
in the dataclass transformer explicitly, both in serialization and deserialization of dataclasses. This affects any type that includes a non-trivial transformation (e.g. FlyteFile
).
Tracking Issue
flyteorg/flyte#3187
Follow-up issue
NA
flyteorg/flytekit
✅ All checks have passed
1/1 successful checksGitHub
12/24/2022, 3:53 PMStructuredDataset
and then trying to load them into a polars dataframe and a hugging face dataset.
It resulted in the following error for both plugins No such file or directory: /var/folders/wq/3hjh3ms916b6dj56zx0f_x000000gq/T/flyte-69d2tww2/sandbox/local_flytekit/95bac8efeb64a8d10d34c73b66df7051/00000
. However, it did work for pandas.
It seems like polars and huggingface add in 00000
to the path in the transformers and spark does not.
• polars: https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-polars/flytekitplugins/polars/sd_transformers.py#L43
• spark: https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-spark/flytekitplugins/spark/sd_transformers.py#L29
Expected behavior
I would expect to be able to use a StructuredDataset
from spark with dataframe libraries from all plugins.
Additional context to reproduce
from flytekit import task, StructuredDataset
from flytekitplugins.spark.task import Spark
from datasets import Dataset
import polars as pl
import datasets
import pandas as pd
@task(
task_config=Spark()
)
def spark_task(path: str) -> StructuredDataset:
sess = flytekit.current_context().spark_session
df = sess.read.parquet(path)
return StructuredDataset(dataframe=df)
df = spark_task(path="./ratings_100k.parquet")
try:
df.open(pl.DataFrame).all().head()
except Exception as e:
print(e)
try:
df.open(datasets.Dataset).all().head()
except Exception as e:
print(e)
df.open(pd.DataFrame).all().head()
Screenshots
Screen Shot 2022-12-24 at 10 52 29 AM▾
GitHub
12/25/2022, 1:04 AMGitHub
12/25/2022, 2:30 AM<https://github.com/flyteorg/flytekit/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flytekit/commit/242a11cf1fe564d6310eda4f1df4f8b7cb8f334a|242a11cf>
- Handle Optional[FlyteFile] in Dataclass type transformer (#1393)
flyteorg/flytekitGitHub
12/26/2022, 7:21 AM{"asctime": "2022-12-25 22:57:07,317", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.PandasToParquetEncodingHandler object at 0x7fce624a8220> with format parquet as default for <class 'pandas.core.frame.DataFrame'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,317", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.PandasToParquetEncodingHandler object at 0x7fce624a8220> with format parquet as default for <class 'pandas.core.frame.DataFrame'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,317", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.PandasToParquetEncodingHandler object at 0x7fce624a8220> with format parquet as default for <class 'pandas.core.frame.DataFrame'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,317", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.PandasToParquetEncodingHandler object at 0x7fce624a8220> with format parquet as default for <class 'pandas.core.frame.DataFrame'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,317", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ParquetToPandasDecodingHandler object at 0x7fce624549a0> with format parquet as default for <class 'pandas.core.frame.DataFrame'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,317", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ParquetToPandasDecodingHandler object at 0x7fce624549a0> with format parquet as default for <class 'pandas.core.frame.DataFrame'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,317", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ParquetToPandasDecodingHandler object at 0x7fce624549a0> with format parquet as default for <class 'pandas.core.frame.DataFrame'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,317", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ParquetToPandasDecodingHandler object at 0x7fce624549a0> with format parquet as default for <class 'pandas.core.frame.DataFrame'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,317", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ParquetToPandasDecodingHandler object at 0x7fce624549a0> with format parquet as default for <class 'pandas.core.frame.DataFrame'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,317", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ArrowToParquetEncodingHandler object at 0x7fce62454940> with format parquet as default for <class 'pyarrow.lib.Table'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,318", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ArrowToParquetEncodingHandler object at 0x7fce62454940> with format parquet as default for <class 'pyarrow.lib.Table'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,318", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ArrowToParquetEncodingHandler object at 0x7fce62454940> with format parquet as default for <class 'pyarrow.lib.Table'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,318", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ArrowToParquetEncodingHandler object at 0x7fce62454940> with format parquet as default for <class 'pyarrow.lib.Table'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,318", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ParquetToArrowDecodingHandler object at 0x7fce624a82b0> with format parquet as default for <class 'pyarrow.lib.Table'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,318", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ParquetToArrowDecodingHandler object at 0x7fce624a82b0> with format parquet as default for <class 'pyarrow.lib.Table'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,318", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ParquetToArrowDecodingHandler object at 0x7fce624a82b0> with format parquet as default for <class 'pyarrow.lib.Table'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,318", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ParquetToArrowDecodingHandler object at 0x7fce624a82b0> with format parquet as default for <class 'pyarrow.lib.Table'>, parquet already specified."}
{"asctime": "2022-12-25 22:57:07,318", "name": "flytekit", "levelname": "WARNING", "message": "Not using handler <flytekit.types.structured.basic_dfs.ParquetToArrowDecodingHandler object at 0x7fce624a82b0> with format parquet as default for <class 'pyarrow.lib.Table'>, parquet already specified."}
Type
☑︎ Bug Fix
☐ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☐ Smoke tested
☐ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
^^^
Tracking Issue
NA
Follow-up issue
NA
flyteorg/flytekitGitHub
12/26/2022, 8:26 AMipywidgets
module not found
[0]: code:"UnknownError" message:"Pod failed. No message received from kubernetes.\r\n[f773f1ce96ea54485bd2-n0-0-0] terminated with exit code (1). Reason [Error]. Message: \nandler <flytekit.types.structured.basic_dfs.ParquetToArrowDecodingHandler object at 0x7f29ae5f4fa0> with format parquet as default for <class 'pyarrow.lib.Table'>, parquet already specified.\"}\n{\"asctime\": \"2022-12-26 06:20:50,244\", \"name\": \"flytekit\", \"levelname\": \"WARNING\", \"message\": \"Not using handler <flytekit.types.structured.basic_dfs.ParquetToArrowDecodingHandler object at 0x7f29ae5f4fa0> with format parquet as default for <class 'pyarrow.lib.Table'>, parquet already specified.\"}\nTraceback (most recent call last):\n File \"/usr/local/bin/pyflyte-fast-execute\", line 5, in <module>\n from flytekit.bin.entrypoint import fast_execute_task_cmd\n File \"/usr/local/lib/python3.9/site-packages/flytekit/__init__.py\", line 261, in <module>\n load_implicit_plugins()\n File \"/usr/local/lib/python3.9/site-packages/flytekit/__init__.py\", line 257, in load_implicit_plugins\n p.load()\n File \"/usr/local/lib/python3.9/site-packages/importlib_metadata/__init__.py\", line 208, in load\n module = import_module(match.group('module'))\n File \"/usr/local/lib/python3.9/importlib/__init__.py\", line 127, in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n File \"/usr/local/lib/python3.9/site-packages/flytekitplugins/deck/__init__.py\", line 15, in <module>\n from .renderer import BoxRenderer, FrameProfilingRenderer, MarkdownRenderer\n File \"/usr/local/lib/python3.9/site-packages/flytekitplugins/deck/renderer.py\", line 4, in <module>\n from pandas_profiling import ProfileReport\n File \"/usr/local/lib/python3.9/site-packages/pandas_profiling/__init__.py\", line 6, in <module>\n from pandas_profiling.compare_reports import compare\n File \"/usr/local/lib/python3.9/site-packages/pandas_profiling/compare_reports.py\", line 7, in <module>\n from pandas_profiling.profile_report import ProfileReport\n File \"/usr/local/lib/python3.9/site-packages/pandas_profiling/profile_report.py\", line 9, in <module>\n from ipywidgets.widgets.widget_box import VBox\nModuleNotFoundError: No module named 'ipywidgets'\n."
image▾