GitHub
03/24/2023, 4:28 AMlist
transformer and then recursively find a pickle transformer. Also due the default pickle transport is a File, a list of even simple objects can result in 100's of files and since they are asynchronously written can cause a huge performance penalty.
Example
import attr
import flytekit
## Worflow below is slow
@attr.define
class Foo:
x: int
@flytekit.task
def t1_slow() -> list[Foo]:
return [Foo(i) for i in range(2000)]
@flytekit.task
def t2_slow(foos: list[Foo]) -> list[int]:
return [foo.x for foo in foos]
@flytekit.workflow
def wf_slow() -> list[int]:
foos = t1_slow()
return t2_slow(foos=foos)
## Worflow below is fast
@attr.define
class Container:
foos: list[Foo]
@flytekit.task
def t1_fast() -> Container:
return Container([Foo(i) for i in range(2000)])
@flytekit.task
def t2_fast(container: Container) -> list[int]:
return [foo.x for foo in container.foos]
@flytekit.workflow
def wf_fast() -> list[int]:
container = t1_fast()
return t2_fast(container=container)
For context, wf_slow can be around 30x slower in some random cases.
Goal: What should the final outcome look like, ideally?
In both the cases the performance should be close to possible. For pickle objects in general that are small, we could use a binary transport?
Describe alternatives you've considered
NA
Propose: Link/Inline OR Additional context
In pickle transformer we can add the following snippet
https://github.com/flyteorg/flytekit/blob/de2878903b594069f1629d7ee11bf59ab04b59f2/flytekit/types/pickle/pickle.py#L51
if lv.scalar.blob:
... # continue with current logic
elif lv.scalar.binary:
... # load bytearray
and while to_literal
x = cloudpickle.dumps()
if len(x) < threshold:
lv.scalar.binary = x
else
continue
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
03/24/2023, 4:35 AM<https://github.com/flyteorg/flyte/tree/master|master>
by jeevb
<https://github.com/flyteorg/flyte/commit/27ae9e0b206575ead777619acc515835f4858fed|27ae9e0b>
- Add support for updating flytecopilot image in flyte-binary helm chart (#3526)
flyteorg/flyteGitHub
03/24/2023, 7:32 AMtf.keras.Model
to be utilized as a Flyte type.
Type
☐ Bug Fix
☐ Feature
☑︎ Plugin
Are all requirements met?
☑︎ Code completed
☐ Smoke tested
☑︎ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
How did you fix the bug, make the feature etc. Link to any design docs etc
Tracking Issue
flyteorg/flyte#2570
Follow-up issue
NA
OR
https://github.com/flyteorg/flyte/issues/
flyteorg/flytekit
GitHub Actions: build-plugins (3.8, flytekit-onnx-pytorch)
GitHub Actions: build-plugins (3.8, flytekit-modin)
GitHub Actions: build-plugins (3.8, flytekit-kf-tensorflow)
GitHub Actions: build-plugins (3.8, flytekit-kf-pytorch)
GitHub Actions: build-plugins (3.8, flytekit-kf-mpi)
GitHub Actions: build-plugins (3.8, flytekit-k8s-pod)
GitHub Actions: build-plugins (3.8, flytekit-hive)
GitHub Actions: build-plugins (3.8, flytekit-greatexpectations)
GitHub Actions: build-plugins (3.8, flytekit-duckdb)
GitHub Actions: build-plugins (3.8, flytekit-dolt)
GitHub Actions: build-plugins (3.8, flytekit-deck-standard)
GitHub Actions: build-plugins (3.8, flytekit-dbt)
GitHub Actions: build-plugins (3.8, flytekit-data-fsspec)
GitHub Actions: build-plugins (3.8, flytekit-dask)
GitHub Actions: build-plugins (3.8, flytekit-bigquery)
GitHub Actions: build-plugins (3.8, flytekit-aws-sagemaker)
GitHub Actions: build-plugins (3.8, flytekit-aws-batch)
GitHub Actions: build-plugins (3.8, flytekit-aws-athena)
GitHub Actions: docs
GitHub Actions: build (windows-latest, 3.9)
GitHub Actions: build (windows-latest, 3.8)
GitHub Actions: build (ubuntu-latest, 3.10)
GitHub Actions: build (ubuntu-latest, 3.9)
GitHub Actions: build (ubuntu-latest, 3.8)
GitHub Actions: Docs Warnings
GitHub Actions: lint
✅ 2 other checks have passed
2/28 successful checksGitHub
03/24/2023, 9:49 AMpip-compile
commands to generate the new requirements
files, it failed on protobuf
version error
Type
☐ Bug Fix
☑︎ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☑︎ Smoke tested
☑︎ Unit tests added
☑︎ Code documentation added
☐ Any pending items have an associated Issue
Complete description
How did you fix the bug, make the feature etc. Link to any design docs etc
Tracking Issue
flyteorg/flyte#2570
Follow-up issue
NA
flyteorg/flytekit
✅ All checks have passed
2/2 successful checksGitHub
03/24/2023, 10:10 AM<https://github.com/flyteorg/flytekit-java/tree/master|master>
by andresgomezfrr
<https://github.com/flyteorg/flytekit-java/commit/abcf4db44feeb41b175e66bfe22c6e66e36ba4c9|abcf4db4>
- Add support to apply node using a Map[String, SdkBindingData[_]] (#216)
flyteorg/flytekit-javaGitHub
03/24/2023, 11:25 AM<https://github.com/flyteorg/flytekit-java/tree/master|master>
by github-actions[bot]
<https://github.com/flyteorg/flytekit-java/commit/dea408199370f3d3eb170b98eaeb547e815600ed|dea40819>
- [maven-release-plugin] prepare release 0.4.13
flyteorg/flytekit-javaGitHub
03/24/2023, 11:25 AM<https://github.com/flyteorg/flytekit-java/tree/master|master>
by github-actions[bot]
<https://github.com/flyteorg/flytekit-java/commit/970b2f58f856b702a206a6b03533c213d0ce46a1|970b2f58>
- [maven-release-plugin] prepare for next development iteration
flyteorg/flytekit-javaGitHub
03/24/2023, 12:59 PMGitHub
03/24/2023, 12:59 PMGitHub
03/24/2023, 1:01 PMGitHub
03/24/2023, 4:25 PM<https://github.com/flyteorg/flytepropeller/tree/master|master>
by hamersaw
<https://github.com/flyteorg/flytepropeller/commit/e4624120c85c0c3ebf5a93f75ae57065b306e4a0|e4624120>
- Add reported_at timestamps to events (#529)
flyteorg/flytepropellerGitHub
03/24/2023, 5:20 PMGitHub
03/24/2023, 5:45 PMGitHub
03/24/2023, 6:04 PM<https://github.com/flyteorg/flytepropeller/tree/master|master>
by hamersaw
<https://github.com/flyteorg/flytepropeller/commit/b1e54821c98093f75e4ab19a9545d3b704ac6d99|b1e54821>
- Including all upstream node deps on BranchNode subnode execution (#543)
flyteorg/flytepropellerGitHub
03/24/2023, 6:13 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by hamersaw
<https://github.com/flyteorg/flytekit/commit/36fe5813d83477f91056f61bd234e9dd232ec4a2|36fe5813>
- add pod_template and pod_template_name arguments for ContainerTask (#1515)
flyteorg/flytekitGitHub
03/24/2023, 6:37 PM<Skeleton />
components where appropriate. This is probably one of those cases, as we don't want a succession of various loading indicators flashing into different spots in the form.
• Loading indicators should only show if requests take longer than 1 second.
• Errors should use a small or medium variant of the DataError
component.
flyteorg/flyteconsoleGitHub
03/24/2023, 6:37 PMreact-query
instead of our custom fetching hook. While it's not hurting anything to continue using the old fetching logic for other parts of the application, maintaining two different patterns creates confusion and more cognitive load for contributors. This task tracks the complete deprecation and removal of useFetchableData
in favor of react-query
.
Flyte component
☐ Overall
☐ Flyte Setup and Installation scripts
☐ Flyte Documentation
☐ Flyte communication (slack/email etc)
☐ FlytePropeller
☐ FlyteIDL (Flyte specification language)
☐ Flytekit (Python SDK)
☐ FlyteAdmin (Control Plane service)
☐ FlytePlugins
☐ DataCatalog
☐ FlyteStdlib (common libraries)
☑︎ FlyteConsole (UI)
☐ Other
flyteorg/flyteGitHub
03/24/2023, 6:38 PMGitHub
03/24/2023, 6:39 PMScreen Shot 2020-12-10 at 9 46 25 AM▾
GitHub
03/24/2023, 6:41 PMGitHub
03/24/2023, 6:42 PMGitHub
03/24/2023, 6:43 PMGitHub
03/24/2023, 6:49 PMGitHub
03/24/2023, 6:50 PMGitHub
03/24/2023, 6:51 PMimage▾
GitHub
03/24/2023, 6:51 PMimage▾
image▾
image▾
GitHub
03/24/2023, 7:41 PMGitHub
03/24/2023, 8:17 PMnoop
task to skip the branch. This will lead to waste of time during execution.
For example:
@task
def noop():
...
conditional("generate_artifact")
.if_(need_to_generate.is_true())
.then(generate_artifact())
.else_(noop())
This makes the code ugly and UI messy. We will need to figure out a way to cleanly handle this.
Goal: What should the final outcome look like, ideally?
Ideally, we can introduce something like`end()` or skip()
operation in conditional to skip the branch, similar to fail()
operation here: https://github.com/flyteorg/flytekit/blob/master/flytekit/core/condition.py#L295
Describe alternatives you've considered
noop() task with cache is being used right now. it is bad coding practice and confusing in the UI.
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
03/24/2023, 8:18 PMalpine:3.15
, as this image has only 1 known vulnerabilities. To do this, merge this pull request, then verify your application still works as expected.
Some of the most important vulnerabilities in your base image include:
* * *
Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information: https://camo.githubusercontent.com/b2f3d3f1a44598de62b980a6dae389b2a137e646ac357e0eb6db13387aeed1dc/68747470733a2f2f6170692e7365676d656e742e696f2f76312f706978656c2f747261636b3f646174613d65794a33636d6c305a55746c65534936496e4a79576d785a634564485932527954485a7362306c596430645563566734576b4652546e4e434f5545774969776959573576626e6c746233567a535751694f694a6c597a497a4d6d49304e79303259574a694c5452694f4463744f5467354f4331685a6a4d345a6d5933595745304e6d59694c434a6c646d567564434936496c425349485a705a58646c5a434973496e42796233426c636e52705a584d694f6e736963484a4a5a434936496d566a4d6a4d79596a51334c545a68596d49744e4749344e7930354f446b344c57466d4d7a686d5a6a6468595451325a694a3966513d3d
🧐 View latest project report
🛠️ Adjust project settings
* * *
Learn how to fix vulnerabilities with free interactive lessons:
🦉 Learn about vulnerability in an interactive lesson of Snyk Learn.
flyteorg/flyteadmin
DCO: DCO
✅ 1 other check has passed
1/2 successful checksGitHub
03/24/2023, 8:31 PM<https://github.com/flyteorg/flyteconsole/tree/master|master>
by ursucarina
<https://github.com/flyteorg/flyteconsole/commit/6816a5f9e8c8b3844f1084496553cd6f65290280|6816a5f9>
- chore: add build:watch to all packages (#731)
flyteorg/flyteconsole