GitHub
06/13/2023, 8:35 PMGitHub
06/13/2023, 10:30 PMkubeflow.pytorch
plugin instead of legacy pytorch
plugin in flyteorg/flytekit#1678 by @fg91
• More time info for time line deck in flyteorg/flytekit#1680 by @Yicheng-Lu-llll
Flyteadmin
• Add logs and stats to sync loop in flyteorg/flyteadmin#573 by Haytham Abuelfutuh
• Fix utf-8 encoding issues with trimmed error messages in flyteorg/flyteadmin#569 by Katrina Rogan
• Tiny url improvements in flyteorg/flyteadmin#565 by Yee Hing Tong
• Update startedAt timestamp only if not set in flyteorg/flyteadmin#567 pmahindrakar-oss
• Save execution namespace in system metadata in flyteorg/flyteadmin#568 by Katrina Rogan
Flyteconsole
• fix: navlink in flyteorg/flyteconsole#772 by 4nalog
• fix: bump version for console in flyteorg/flyteconsole#769 by 4nalog
• fix: preserve domain when navigating using sidebar in flyteorg/flyteconsole#768 by 4nalog
• fix: dynamic-node-tasks in flyteorg/flyteconsole#765 by 4nalog
• chore: hide map task runtime info in https://github.com/flyteorg/flyteconsole7/66 by Carina Ursu
• Bug: union val missing for LP relaunch in flyteorg/flyteconsole#762 by Frank Flitton
• Feature: Fullview Flyte Deck modal in flyteorg/flyteconsole#764 by Frank Flitton
• chore: add item when mapped task in flyteorg/flyteconsole#761 by Jason Porter
• Bug: Execution Page's back button returns Workflows route from Launch Plan route https://github.com/flyteorg/flyteconsolepatch in flyteorg/flyteconsole#760 by Frank Flitton
Flytepropeller
• Register gRPC plugin after reading configmap in flyteorg/flytepropeller#564 by Kevin Su
• Not stripping structure from literal types in flyteorg/flytepropeller#571 by Dan Rammer
• Bump flyteplugins to v1.0.63 in flyteorg/flytepropeller#568 by bstadlbauer
• bumped flyteplugins in flyteorg/flytepropeller#566 by Dan Rammer
• Use correct k8 client in flyteorg/flytepropeller#563 by sonjaer
flyteorg/flyteGitHub
06/13/2023, 10:32 PMGitHub
06/14/2023, 12:25 AM<https://github.com/flyteorg/flytekit/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flytekit/commit/a68cb34799237f6fd52396013b11bfcf02fbab7d|a68cb347>
- upload deck.html only with deck enable (#1693)
flyteorg/flytekitGitHub
06/14/2023, 1:00 AM<https://github.com/flyteorg/flytekit/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flytekit/commit/9977aac26242ebbede8e00d476c2fbc59ac5487a|9977aac2>
- Local agent (#1687)
flyteorg/flytekitGitHub
06/14/2023, 2:28 AMimage▾
image▾
image▾
GitHub
06/14/2023, 3:46 AMGitHub
06/14/2023, 4:05 AMkubeflow.pytorch
plugin instead of legacy pytorch
plugin by @fg91 in #1678
• More time info for time line deck by @Yicheng-Lu-llll in #1680
• Rename external plugin to agent by @pingsutw in #1666
• feat: Add Auth0/audience support for ClientCredentials flow by @PudgyPigeon in #1639
• pyflyte run remote file by @ChungYujoyce in #1670
• Upload 'deck.html' only when the deck is enabled by @Yicheng-Lu-llll in #1693
• Local agent by @pingsutw in #1687
New Contributors
• @yubofredwang made their first contribution in #1627
• @PudgyPigeon made their first contribution in #1639
• @ChungYujoyce made their first contribution in #1670
Full Changelog: v1.6.2...v1.7.0
flyteorg/flytekitGitHub
06/14/2023, 4:26 AMimage▾
GitHub
06/14/2023, 4:46 PM<https://github.com/flyteorg/flyte/tree/master|master>
by katrogan
<https://github.com/flyteorg/flyte/commit/bdce6e6f0afeeb62acb2173649f5d69820d6f9b2|bdce6e6f>
- Allow setting resources for cluster resource controller deployment (#3762)
flyteorg/flyteGitHub
06/14/2023, 5:26 PM[SECURITY] v1.2.26
```
v1.2.26 - 14 Jun 2023
[Security]
* Potential Padding Oracle Attack Vulnerability and Timing Attack Vulnerability
for JWE AES-CBC encrypted payloads affecting all v2 releases up to v2.0.10,
all v1 releases up to v1.2.25, and all v0 releases up to v0.9.2 have been reported by
@shogo82148.
Please note that v0 versions will NOT receive fixes.
This release fixes these vulnerabilities for the v1 series.
```
v1.2.25
```
v1.2.25 23 May 2022
[Bug Fixes][Security]
* [jwe] An old bug from at least 7 years ago existed in handling AES-CBC unpadding,
where the unpad operation might remove more bytes than necessary ([#744](https://github.com/lestrrat-go/jwx/issues/744))
This affects all jwx code that is available before v2.0.2 and v1.2.25.
```
v1.2.24
```
v1.2.24 05 May 2022
[Security]
* Upgrade golang.org/x/crypto ([#724](https://github.com/lestrrat-go/jwx/issues/724))
```
v1.2.23
```
v1.2.23 13 Apr 2022
[Bug fixes]
* [jwk] jwk.AutoRefresh had a race condition whenwasConfigure()
called concurrently ([#686](https://github.com/lestrrat-go/jwx/issues/686))
(It has been patched correctly, but we may come back to revisit
the design choices in the near future)
```
v1.2.22
```
v1.2.22 08 Apr 2022
[Bug fixes]
* [jws] jws.Verify was ignoring theheader when it was presentb64
in the protected headers ([#681](https://github.com/lestrrat-go/jwx/issues/681)). Now the following should work:
jws.Sign(..., jws.WithDetachedPayload(payload))
// previously payload had to be base64 encoded
jws.Verify(..., jws.WithDetachedPayload(payload))
```... (truncated) Changelog Sourced from github.com/lestrrat-go/jwx's changelog.
v1.2.26 - 14 Jun 2023 [Security]
• Potential Padding Oracle Attack Vulnerability and Timing Attack Vulnerability for JWE AES-CBC encrypted payloads affecting all v2 releases up to v2.0.10, all v1 releases up to v1.2.25, and all v0 releases up to v0.9.2 have been reported by `@shogo82148`.
Please note that v0 versions will NOT receive fixes. This release fixes these vulnerabilities for the v1 series.
[Miscellaneous]
• JWE tests now only run algorithms that are supported by the underlyingtooljose
v1.2.25 23 May 2022 [Bug Fixes][Security]
• [jwe] An old bug from at least 7 years ago existed in handling AES-CBC unpadding, where the unpad operation might remove more bytes than necessary (#744) This affects all jwx code that is available before v2.0.2 and v1.2.25.
v1.2.24 05 May 2022 [Security]
• Upgrade golang.org/x/crypto (#724)
v1.2.23 13 Apr 2022 [Bug fixes]
• [jwk] jwk.AutoRefresh had a race condition whenwas called concurrently (#686) (It has been patched correctly, but we may come back to revisit the design choices in the near future)Configure()
v1.2.22 08 Apr 2022 [Bug fixes]
• [jws] jws.Verify was ignoring theheader when it was present in the protected headers (#681). Now the following should work:b64
jws.Sign(..., jws.WithDetachedPayload(payload)) // previously payload had to be base64 encoded jws.Verify(..., jws.WithDetachedPayload(payload))
(note: v2 branch was not affected)
v1.2.21 30 Mar 2022 [Bug fixes]
• [jwk] RSA keys without p and q can now be parsed.
v1.2.20 03 Mar 2022 [Miscellaneous]
• Dependency on golang.org/x/crypto has been upgraded to v0.0.0-20220214200702-86341886e292 to address... (truncated) Commits • `d9ddbc8` merge v1 (#936) • `ad8c29d` merge develop/v1 (#747) • `e38f677` Merge develop/v1 (#727) • `baba561` Merge branch 'develop/v1' into v1 • `8ff6c75` Update Changes • `ea97e8c` Fix race in
jwk.AutoRefresh
(#686)
• `f4701e1` Update Changes
• `e831228` Fix jws.Verify not respecting the b64 header in the protected headers (#683)
• `b66a2cb` backport: Update golangci lint (#679) (#680)
• `4899c32` reword error
• Additional commits viewable in compare view
Dependabot compatibility score
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase
.
* * *
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
• @dependabot rebase
will rebase this PR
• @dependabot recreate
will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge
will merge this PR after your CI passes on it
• @dependabot squash and merge
will squash and merge this PR after your CI passes on it
• @dependabot cancel merge
will cancel a previously requested merge and block automerging
• @dependabot reopen
will reopen this PR if it is closed
• @dependabot close
will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version
will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version
will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency
will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the Security Alerts page.
flyteorg/flyteadmin
GitHub Actions: End2End Test / End to End tests
GitHub Actions: Integration Test / Integration tests
✅ 8 other checks have passed
8/10 successful checksGitHub
06/14/2023, 6:41 PMGitHub
06/14/2023, 6:47 PM<https://github.com/flyteorg/flyte/tree/master|master>
by iaroslav-ciupin
<https://github.com/flyteorg/flyte/commit/201a8e1450d86b72a67be938ce7502ee2621cadb|201a8e14>
- adjust MPI operator installation doc (#3781)
flyteorg/flyteGitHub
06/14/2023, 6:58 PMimage▾
GitHub
06/14/2023, 7:24 PMimage▾
GitHub
06/14/2023, 7:25 PMGetFlyteKitMetrics
.
image▾
GitHub
06/14/2023, 8:10 PM<https://github.com/flyteorg/flyteplugins/tree/master|master>
by hamersaw
<https://github.com/flyteorg/flyteplugins/commit/318fa6b0e86495c144517f2a852362a31765bdc6|318fa6b0>
- fix v1 pytorch job plugin with elastic policy (#359)
flyteorg/flytepluginsGitHub
06/14/2023, 8:11 PMGitHub
06/14/2023, 10:16 PMtimestamp >= podStartTime
to our stackdriver log link template. This is important because currently our users have to manually extend the time range (default 1h) each time they visit the stackdriver log page. (Unless the job started and finished within the past hour.)
To this aim, I recently made the pod start and finish time available as logging link template variables in RFC3339 time format which is required by google cloud logging (#360).
Currently, the kubeflow plugin doesn't pass the pod start time to the log plugin though.
In this PR I extend the kubeflow plugin to do so.
Type
☐ Bug Fix
☑︎ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☑︎ Smoke tested
☑︎ Unit tests added
☐ Code documentation added
☑︎ Any pending items have an associated Issue
Complete description
NA
Tracking Issue
NA
Follow-up issue
NA
flyteorg/flyteplugins
✅ All checks have passed
7/7 successful checksGitHub
06/14/2023, 10:53 PM32.0.0
Maven
```
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>32.0.0-jre</version>
<!-- or, for Android: -->
<version>32.0.0-android</version>
</dependency>
```
Jar files
• 32.0.0-jre.jar
• 32.0.0-android.jar
Guava requires one runtime dependency, which you can download here:
• failureaccess-1.0.1.jar
Javadoc
• 32.0.0-jre
• 32.0.0-android
JDiff
• 32.0.0-jre vs. 31.1-jre
• 32.0.0-android vs. 31.1-android
• 32.0.0-android vs. 32.0.0-jre
Changelog
Security fixes
• ReimplementedandFiles.createTempDir
to further address CVE-2020-8908 (#4011) and CVE-2023-2976 (#2575). (feb83a1c8f)FileBackedOutputStream
While CVE-2020-8908 was officially closed when we deprecatedin Guava 30.0, we've heard from users that even recent versions of Guava have been listed as vulnerable in other databases of security vulnerabilities. In response, we've reimplemented the method (and the very rarely usedFiles.createTempDir
class, which had a similar issue) to eliminate the insecure behavior entirely. This change could technically affect users in a number of different ways (discussed under "Incompatible changes" below), but in practice, the only problem users are likely to encounter is with Windows. If you are using those APIs under Windows, you should skip 32.0.0 and go straight to 32.0.1 which fixes the problem. (Unfortunately, we didn't think of the Windows problem until after the release. And while we warn that `common.io` in particular may not work under Windows, we didn't intend to regress support.) Sorry for the trouble.FileBackedOutputStream
Incompatible changes
Although this release bumps Guava's major version number, it makes no binary-incompatible changes to theartifact.guava
One change could cause issues for Widows users, and a few other changes could cause issues for users in more usual situations:
• The new implementations ofandFiles.createTempDir
throw an exception under Windows. This is fixed in 32.0.1. Sorry for the trouble.FileBackedOutputStream
• This release makes a binary-incompatible change to aAPI in the separate artifact@Beta
. Specifically, we changed the return type ofguava-testlib
toTestingExecutors.sameThreadScheduledExecutor
. The old return type was a package-private class, which caused the Kotlin compiler to produce warnings. (dafaa3e435)ListeningScheduledExecutorService
• This release adds two methods to the Android flavor of Guava:... (truncated) Commits • See full diff in compare view Dependabot compatibility score Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commentingandInvokable.getAnnotatedReturnType()
. Those methods do not work under an Android VM; we added them only to help our tests of the Android flavor (since we also run those tests under a JRE). Android VMs tolerate such methods as long as the app does not call them or perform reflection on them, and builds tolerate them because of our new Proguard configurations (discussed below). Thus, we expect no impact to most users. However, we could imagine build problems for users who have set up their own build system for the Android flavor of Guava. Please report any problems so that we can judge how safely we might be able to add other methods to the Android flavor in the future, such as APIs that use Java 8 classes likeParameter.getAnnotatedType()
. (b30e73cfa81ad15c1023c17cfd083255a3df0105)Stream
@dependabot rebase
.
* * *
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
• @dependabot rebase
will rebase this PR
• @dependabot recreate
will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge
will merge this PR after your CI passes on it
• @dependabot squash and merge
will squash and merge this PR after your CI passes on it
• @dependabot cancel merge
will cancel a previously requested merge and block automerging
• @dependabot reopen
will reopen this PR if it is closed
• @dependabot close
will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version
will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version
will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency
will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the Security Alerts page.
flyteorg/flytekit-java
GitHub Actions: build-snapshot
✅ 2 other checks have passed
2/3 successful checksGitHub
06/15/2023, 2:03 AMGitHub
06/15/2023, 3:20 AM<https://github.com/flyteorg/flytekit/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytekit/commit/207d01986d744f8212a3e2e28826ce947efa96d9|207d0198>
- Support configuring pip index url with the image spec (#1692)
flyteorg/flytekitGitHub
06/15/2023, 4:23 AM<https://github.com/flyteorg/flytesnacks/tree/master|master>
by samhita-alla
<https://github.com/flyteorg/flytesnacks/commit/374d29e89c12142b24ba0fd5aab3f0f4b4a9a3e6|374d29e8>
- Image spec example (#988)
flyteorg/flytesnacksGitHub
06/15/2023, 6:03 AMGitHub
06/15/2023, 1:51 PMflytekit==1.6.0
rich tracebacks were introduced (#1582).
Before, exceptions were shown like this:
Screenshot 2023-06-15 at 15 37 04▾
Screenshot 2023-06-15 at 15 36 44▾
import flytekit
is called somewhere in the imported code base, also when not directly working with flyte.
* * *
Style ultimately is a question of taste but several of our engineers have complained about this change:
When/Why did the stacktrace get souglycolorful. It takes up way too much space and pycharm cannot create crosslinks to the files anymore for me 🙈 And long lines are truncated
I feel the same, does not help me
Can we configure this
I wanted to complain about this too 😄
The tracebacks contain way more (white space) characters, meaning I can paste a lot less of them into chat gptBecause of this, in this PR I propose to introduce an environment variable which disables rich tracebacks. I chose a naming inspired by
"FLYTE_SDK_LOGGING_LEVEL"
or "FLYTE_SDK_LOGGING_FORMAT"
in flytekit/loggers.py
.
Would this change be be acceptable to you @cosmicBboy @Ketan (kumare3) @eapolinario ?
Type
☐ Bug Fix
☑︎ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☑︎ Smoke tested
☐ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
NA
Tracking Issue
NA
Follow-up issue
NA
flyteorg/flytekit
✅ All checks have passed
30/30 successful checksGitHub
06/15/2023, 2:19 PM<https://github.com/flyteorg/flyteplugins/tree/master|master>
by hamersaw
<https://github.com/flyteorg/flyteplugins/commit/dfdf6f95aef7bebff160d6660f5c62f5832c39e4|dfdf6f95>
- Allow using pod start time in kubeflow plugin log links (#362)
flyteorg/flytepluginsGitHub
06/15/2023, 2:19 PMinput_enum
as an enum:
╭───────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────╮
│ /Users/reda/Projects/unionai_bugs/enum_remote/replicate.py:52 in <module> │
│ │
│ ❱ 52 workflow_execution = remote.execute( │
│ │
│ /Users/reda/miniconda3/envs/union_ai/lib/python3.10/site-packages/flytekit/remote/remote.py:1138 in execute │
│ │
│ ❱ 1138 │ │ │ return self.execute_remote_task_lp( │
│ │
│ /Users/reda/miniconda3/envs/union_ai/lib/python3.10/site-packages/flytekit/remote/remote.py:1227 in │
│ execute_remote_task_lp │
│ │
│ ❱ 1227 │ │ return self._execute( │
│ │
│ /Users/reda/miniconda3/envs/union_ai/lib/python3.10/site-packages/flytekit/remote/remote.py:1002 in _execute │
│ │
│ ❱ 1002 │ │ │ │ │ hint = type_hints[k] │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'input_enum'
This is similar to #2278 though the error message is different
Expected behavior
Workflow should execute
Additional context to reproduce
Here is a script to reproduce the bug against the sandbox
import typing
from enum import Enum
import flytekit
from flytekit.remote import FlyteRemote
from flytekit.configuration import (
Config,
ImageConfig,
SerializationSettings,
FastSerializationSettings
)
class MyEnum(Enum):
FOO = "FOO"
BAR = "BAR"
@flytekit.task
def get_enum_value(input_enum: MyEnum) -> str:
return input_enum.value
@flytekit.workflow
def wf(input_enum: MyEnum) -> MyEnum:
return get_enum_value(input_enum=input_enum)
_PROJECT = "flytesnacks"
_DOMAIN = "development"
_VERSION = "abc"
remote = FlyteRemote(config=Config.for_sandbox())
img = ImageConfig.auto_default_image()
fast_serialization_settings = FastSerializationSettings(enabled=True, destination_dir="/root")
serialization_settings = SerializationSettings(
image_config=img,
project=_PROJECT,
domain=_DOMAIN,
version=_VERSION,
fast_serialization_settings=fast_serialization_settings
)
remote.register_workflow(wf, serialization_settings=serialization_settings)
remote_workflow = remote.fetch_workflow(
project=_PROJECT,
domain=_DOMAIN,
version=_VERSION,
name="wf"
)
workflow_execution = remote.execute(
remote_workflow,
inputs={"input_enum": MyEnum.FOO},
project=_PROJECT,
domain=_DOMAIN,
wait=True
)
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
06/15/2023, 2:42 PMGitHub
06/15/2023, 3:48 PMtorch.nn.Module
and flytekit
version >1.5.0.
Flyte uses PickleFile
with type nn.Module
as default when importing tasks/workflows. When registering, it does not recognize the type as nn.Module
but as PickleFile
- throws an error because the types aren't matching.
Example of the error:
# a.py
from flytekit import task
@task
def foo() -> int:
return 2
------------------------------------------------------------------------------
# b.py
from flytekit import task
from torch import nn
@task
def bla() -> nn.Module:
return nn.Module()
Warning message:
WARNING {"asctime": "2023-06-15 17:40:51,991", "name": "flytekit", "levelname": "WARNING", "message": "Unsupported Type <class 'torch.nn.modules.module.Module'> found,
Flyte will default to use PickleFile as the transport. Pickle can only be used to send objects between the exact same version of Python, and we strongly recommend
to use python type that flyte support."}
Error message:
Error 0: Code: MismatchingTypes, Node Id: end-node, Description: Variable [model] (type [blob:<format:"PyTorchModule" > ]) doesn't match expected type [blob:<format:"PythonPickle" > metadata:<fields:<key:"python_class_name" value:<string_value:"Module" > > > ].
Error 1: Code: MismatchingTypes, Node Id: n1, Description: Variable [model] (type [blob:<format:"PythonPickle" > metadata:<fields:<key:"python_class_name" value:<string_value:"Module" > > > ]) doesn't match expected type [blob:<format:"PyTorchModule" > ].
Expected behavior
Should be able to register.
The culprit is most likely `if is_imported("torch")`: in flytekit.core.type_engine
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
06/15/2023, 4:48 PMmap_task
is restricted to single-input single-output tasks. There is interest in multi-input map tasks, and using a @DataClass object is a way to get around it (as long as inputs are JSON-serializable).
For the outputs, a task producing a NamedTuple
cannot be used with map_task, and we have to use @dynamic
instead. This issue suggests to support tasks outputting a`NamedTuple` to be wrapped within map_task
.
Goal: What should the final outcome look like, ideally?
Allow tasks outputting a NamedTuple
to be used within map_task
.
Describe alternatives you've considered
Using @dynamic
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyte