GitHub
07/05/2023, 9:52 PMRelease v1.53.0
This is release 1.53.0 (glockenspiel) of gRPC Core.
For gRPC documentation, see grpc.io. For previous releases, see Releases.
This release contains refinements, improvements, and bug fixes, with highlights listed below.
Core
• xDS: fix crash when removing the last endpoint from the last locality in weighted_target. (#32592)
• filter stack: pass peer name up via recv_initial_metadata batch. (#31933)
• [EventEngine] Add advice against blocking work in callbacks. (#32397)
• [http2] Dont drop connections on metadata limit exceeded. (#32309)
• xDS: reject aggregate cluster with empty cluster list. (#32238)
• Fix Python epoll1 Fork Support. (#32196)
• server: introduce ServerMetricRecorder API and move per-call reporting from a C++ interceptor to a C-core filter. (#32106)
• [EventEngine] Add invalid handle types to the public API. (#32202)
• [EventEngine] Refactoring the EventEngine Test Suite: Part 1. (#32127)
• xDS: fix WeightedClusters total weight handling. (#32134)
C++
• Update minimum MSVC version to 2019. (#32615)
• Use CMake variables for paths in pkg-config files. (#31671)
C#
• Grpc.Tools: Use x86 protoc binaries on arm64 Windows. (#32017)
Python
• Support python 3.11 on aarch64. (#32270)
• Include .pyi file. (#32268)
• De-experimentalize wait-for-ready. (#32143)
• De-experimentalize compression. (#32138)
Ruby
• [ruby]: add pre-compiled binaries for ruby 3.2; drop them for ruby 2.6. (#32089)
Release v1.53.0-pre2
This is a prerelease of gRPC Core 1.53.0 (glockenspiel).
For gRPC documentation, see grpc.io. For previous releases, see Releases.... (truncated) Commits • `358bfb5` Bump version to 1.53.0 (#32685) • `6e1ebe7` Backport: Ensure compatibility with the new custom kokoro win2019 image (#326... • `44a77f6` Backport 1.53: Update minimum MSVC version to 2019 (#32615) • `c11153c` backport to 1.53: xDS: fix crash when removing the last endpoint from the las... • `7c7712a` Bump version to 1.53.0-pre2. (#32545) • `a4017dc` backport to 1.53: [promises] Make Poll<T> its own type, not a variant<> (#32540) • `3f93c16` Fuzzer fix backport to v1.53 (#32511) • `5b244b2` Bump release version to 1.53.0-pre1 (#32428) • `6589340` Bump core version 202302161703 (#32416) • `d49e151` [backoff] Add random early detection classifier (#32354) • Additional commits viewable in compare view Dependabot compatibility score Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting
@dependabot rebase
.
* * *
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
• @dependabot rebase
will rebase this PR
• @dependabot recreate
will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge
will merge this PR after your CI passes on it
• @dependabot squash and merge
will squash and merge this PR after your CI passes on it
• @dependabot cancel merge
will cancel a previously requested merge and block automerging
• @dependabot reopen
will reopen this PR if it is closed
• @dependabot close
will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version
will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version
will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency
will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the Security Alerts page.
flyteorg/flyte-conference-talks
GitHub Actions: build (integration, scipy-2023, 3.9)
GitHub Actions: build (unit, scipy-2023, 3.9)
✅ 2 other checks have passed
2/4 successful checksGitHub
07/05/2023, 10:06 PMGitHub
07/05/2023, 10:33 PM<https://github.com/flyteorg/flytepropeller/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flytepropeller/commit/206013ab02af16d326cea394807a1743eefb82ec|206013ab>
- Fix usage of archive replacements (#586)
flyteorg/flytepropellerGitHub
07/05/2023, 11:05 PMdemo
and sandbox
.
flyteorg/flytectl
GitHub Actions: Goreleaser
GitHub Actions: Generate documentation
GitHub Actions: Bump Version
GitHub Actions: Dry Run Goreleaser
✅ 8 other checks have passed
8/12 successful checksGitHub
07/05/2023, 11:15 PMGitHub
07/05/2023, 11:41 PM<https://github.com/flyteorg/flytectl/tree/master|master>
by jeevb
<https://github.com/flyteorg/flytectl/commit/ac3aa739aa7a499179a298715641aaf9ca836193|ac3aa739>
- Remove flytekit image priming (#412)
flyteorg/flytectlGitHub
07/05/2023, 11:52 PMflyte-binary
and flyte-core
I haven't been able to reproduce the Azure AD config as external auth server, so it is left as a separate section to be addressed in an upcoming PR
Check all the applicable boxes
☑︎ I updated the documentation accordingly.
☑︎ All new and existing tests passed.
☑︎ All commits are signed-off.
Screenshots
Note to reviewers
flyteorg/flyte
✅ All checks have passed
6/6 successful checksGitHub
07/06/2023, 12:09 AMGitHub
07/06/2023, 12:09 AM<https://github.com/flyteorg/homebrew-tap/tree/main|main>
by flyte-bot
<https://github.com/flyteorg/homebrew-tap/commit/1938dd8099ea7cb2da6b6b46a47a80b07ec7e57a|1938dd80>
- Brew formula update for flytectl version v0.6.40
flyteorg/homebrew-tapGitHub
07/06/2023, 12:11 AM<https://github.com/flyteorg/flyteadmin/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flyteadmin/commit/53282fe979b628447e72d0e3f8fd0e2b9235d929|53282fe9>
- remove welcome bot from boilerplate (#460)
flyteorg/flyteadminGitHub
07/06/2023, 12:11 AMGitHub
07/06/2023, 12:34 AMGitHub
07/06/2023, 12:45 AMGitHub
07/06/2023, 2:40 AM<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>
by cosmicBboy
<https://github.com/flyteorg/flyte-conference-talks/commit/96233a12dde770be5834cb6fc1a4ab8434da6fbe|96233a12>
- update scipy-2023 readme, workflows, tests
flyteorg/flyte-conference-talksGitHub
07/06/2023, 4:22 AM<https://github.com/flyteorg/flyte/tree/master|master>
by samhita-alla
<https://github.com/flyteorg/flyte/commit/58a1de6dd4df4765aeccce2008bb69eeee1f87ec|58a1de6d>
- Fix broken links (#3830)
flyteorg/flyteGitHub
07/06/2023, 6:38 AM<https://github.com/flyteorg/flytesnacks/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytesnacks/commit/73840e67cf58ee6c3372c0afb8f15c9a63c54771|73840e67>
- Update Athena example (#1007)
flyteorg/flytesnacksGitHub
07/06/2023, 7:02 AMpyflyte run --remote --tags '["hello", "flyte"]' workflow.py wf
Type
☐ Bug Fix
☑︎ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☑︎ Smoke tested
☐ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
Added a new arg (tags) to execution
Tracking Issue
flyteorg/flyte#3320
Follow-up issue
NA
flyteorg/flytekit
GitHub Actions: build-plugins (3.11, flytekit-sqlalchemy)
GitHub Actions: build-plugins (3.11, flytekit-snowflake)
GitHub Actions: build-plugins (3.11, flytekit-polars)
GitHub Actions: build-plugins (3.11, flytekit-papermill)
GitHub Actions: build-plugins (3.11, flytekit-pandera)
GitHub Actions: build-plugins (3.11, flytekit-kf-tensorflow)
GitHub Actions: build-plugins (3.11, flytekit-kf-pytorch)
GitHub Actions: build-plugins (3.11, flytekit-hive)
GitHub Actions: build-plugins (3.11, flytekit-envd)
✅ 21 other checks have passed
21/30 successful checksGitHub
07/06/2023, 7:59 AMfrom typing import Union
from flytekit import task, workflow
@task()
def my_task(number: Union[int, str]):
print(number)
@workflow
def my_work(number: Union[int, str] = 1):
my_task(number=number)
Registering the workflow works: pylyte register workflow.py
✅.
Running the workflow directly works: pylyte run --remote run workflow.py my_workflow
✅.
Trying to run the registered workflow from the WebUI fails ❌ (see screenshot).
Expected behavior
I expect an error at registration or in the WebUI if the workflow is broken. Currently, the WebUI errors out without a hint.
Additional context to reproduce
No response
Screenshots
The error in the WebUI:
Bildschirmfoto 2023-07-06 um 09 56 33▾
Bildschirmfoto 2023-07-06 um 09 59 07▾
GitHub
07/06/2023, 8:09 AMGitHub
07/06/2023, 8:53 AMSubWorkflow
in Java and Scala carries different meaning for different purpose. So we align the Java one with Scala.
Tracking Issue
NA
Follow-up issue
NA
flyteorg/flytekit-java
✅ All checks have passed
3/3 successful checksGitHub
07/06/2023, 10:09 AM<https://github.com/flyteorg/flytekit-java/tree/master|master>
by honnix
<https://github.com/flyteorg/flytekit-java/commit/54a94b48e52b1a09788eaf527fcb1166dfb8506e|54a94b48>
- Align java examples with scala ones (#236)
flyteorg/flytekit-javaGitHub
07/06/2023, 11:08 AMGitHub
07/06/2023, 1:17 PM<https://github.com/flyteorg/flytesnacks/tree/master|master>
by cosmicBboy
<https://github.com/flyteorg/flytesnacks/commit/80427a51dda0edd78c931077ca69213c444887aa|80427a51>
- more formatting cleanup (#1015)
flyteorg/flytesnacksGitHub
07/06/2023, 1:18 PMGitHub
07/06/2023, 2:25 PM<https://github.com/flyteorg/flytepropeller/tree/master|master>
by hamersaw
<https://github.com/flyteorg/flytepropeller/commit/0cfb0a0ad73ca97036311f4d18d908260bfa601c|0cfb0a0a>
- Added IsFailurePermanent flag on DynamicTaskStatus (#567)
flyteorg/flytepropellerGitHub
07/06/2023, 2:25 PMRuntimeError
, the workflow is retried. According to the docs, it seems like this case is considered non-recoverable and should not be retried.
Expected behavior
Workflow and task fail with no retries
Additional context to reproduce
No response
Screenshots
Traceback (most recent call last):
File "/app/pip_pypi__flytekit/flytekit/exceptions/scopes.py", line 203, in user_entry_point
return wrapped(*args, **kwargs)
File "/app/atomwise/...", line 266, in run_block
block.validate_output(cmpds, result)
File "/app/atomwise/..., line 119, in validate_output
raise RuntimeError(
Message:
<blah>
User error.
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
07/06/2023, 2:25 PMIsFailurePermanent
field on the DynamicTaskStatus
struct to indicate that the failure is permanent, and therefore should not be recovered.
Type
☑︎ Bug Fix
☐ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☑︎ Smoke tested
☐ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
The choice to use a new flag is entirely out of backwards compatibility. Other options include:
(1) Using the ExecutionError in flyteplugins io rather than flyteidl ExecutionError: This would make retrieval of in progress dynamic task status fail when it expects a different error type on the field.
(2) Naming the flag IsRecoverable
vs `isPermanent`: This is difficult because intuition says that we should use IsRecoverable
which adheres more to the Flyte standard. The biggest issue is that dynamics treat all failures as retryable by default now, so this would change the default to be non-recoverable (ie. false
for IsRecoverable
) which changes the default behavior. It is VERY difficult to ensure there will be no regressions.
Tracking Issue
fixes flyteorg/flyte#3606
Follow-up issue
NA
flyteorg/flytepropeller
GitHub Actions: Build & Push Flytepropeller Image
GitHub Actions: Goreleaser
GitHub Actions: Bump Version
✅ 11 other checks have passed
11/14 successful checksGitHub
07/06/2023, 2:31 PMSQLite3Task
.
For example running the following task:
QUERY = "SELECT species, bill_length_mm, bill_depth_mm, flipper_length_mm, body_mass_g FROM penguins"
get_data = SQLite3Task(
name="get_penguins_data",
query_template=QUERY,
output_schema_type=PenquinsDataset,
task_config=SQLite3Config(
uri="<https://datasette-seaborn-demo.datasette.io/penguins.db>"
),
)
Produces an error:
[1/1] currentAttempt done. Last Error: USER::� │
│ /usr/local/lib/python3.9/site-packages/flytekit/exceptions/scopes.py:160 in │
│ system_entry_point │
│ │
│ ❱ 160 │ │ │ │ return wrapped(*args, **kwargs) │
│ │
│ /usr/local/lib/python3.9/site-packages/flytekit/bin/entrypoint.py:362 in │
│ _execute_task │
│ │
│ ❱ 362 │ │ _handle_annotated_task(ctx, _task_def, inputs, output_prefix) │
│ │
│ /usr/local/lib/python3.9/site-packages/flytekit/bin/entrypoint.py:304 in │
│ _handle_annotated_task │
│ │
│ ❱ 304 │ _dispatch_execute(ctx, task_def, inputs, output_prefix) │
│ │
│ /usr/local/lib/python3.9/site-packages/flytekit/bin/entrypoint.py:163 in │
│ _dispatch_execute │
│ │
│ ❱ 163 │ if not task_def.disable_deck: │
╰──────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'ExecutableTemplateShimTask' object has no attribute
'disable_deck'
Expected behavior
The ExecutableTemplateShimTask
class doesn't have a disable_deck
attribute.
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
07/06/2023, 2:35 PMSQlite3Task
. This PR fixes an issue in the execution entrypoint that assumes the all task objects have the disable_deck
attribute.
Type
☑︎ Bug Fix
☐ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☐ Smoke tested
☐ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
Use the getattr
function to get disable_deck
, and return False
if not present.
Tracking Issue
https://github.com/flyteorg/flyte/issues/
flyteorg/flytekit
GitHub Actions: build-plugins (3.8, flytekit-huggingface)
GitHub Actions: build-plugins (3.8, flytekit-hive)
GitHub Actions: build-plugins (3.8, flytekit-greatexpectations)
GitHub Actions: build-plugins (3.8, flytekit-envd)
GitHub Actions: build-plugins (3.8, flytekit-duckdb)
GitHub Actions: build-plugins (3.8, flytekit-dolt)
GitHub Actions: build-plugins (3.8, flytekit-deck-standard)
GitHub Actions: build-plugins (3.8, flytekit-dbt)
GitHub Actions: build-plugins (3.8, flytekit-data-fsspec)
GitHub Actions: build-plugins (3.8, flytekit-dask)
GitHub Actions: build-plugins (3.8, flytekit-bigquery)
GitHub Actions: build-plugins (3.8, flytekit-aws-sagemaker)
GitHub Actions: build-plugins (3.8, flytekit-aws-batch)
GitHub Actions: build-plugins (3.8, flytekit-aws-athena)
GitHub Actions: lint
GitHub Actions: build (windows-latest, 3.11)
GitHub Actions: build (windows-latest, 3.9)
GitHub Actions: build (windows-latest, 3.8)
GitHub Actions: build (ubuntu-latest, 3.11)
GitHub Actions: build (ubuntu-latest, 3.10)
GitHub Actions: build (ubuntu-latest, 3.9)
GitHub Actions: build (ubuntu-latest, 3.8)
GitHub Actions: docs
GitHub Actions: Docs Warnings
✅ 2 other checks have passed
2/26 successful checksGitHub
07/06/2023, 2:50 PMdef to_literal(...) -> Literal:
local_path = ...
# Save object to local path
remote_path = ctx.file_access.get_random_remote_path(local_path)
ctx.file_access.put_data(local_path, remote_path, is_multipart=False)
# Return Literal containing remote_path
When objects of such types are passed to @dynamic
workflows and then passed along to tasks called within the dynamic workflow, this behaviour always leads to cache misses. The reason is that in the dynamic workflow, the objects are deserialized and then again serialized to a different random remote path.
Expected behavior
There should not be cache misses in this situation.
Additional context to reproduce
Let us consider this example workflow:
import torch.nn as nn
from flytekit import task, workflow, dynamic
@task(cache=True, cache_version="0.1")
def train(model: nn.Module) -> nn.Module:
print(f"Training model {model}")
return model
@task(cache=True, cache_version="0.1")
def other_task(param: int) -> int:
print(f"Doing something else with param {param}")
return param
@dynamic(cache=True, cache_version="0.1")
def sub_wf(model: nn.Module, param: int) -> tuple[nn.Module, int]:
other_task(param=param)
train(model=model)
return model, param
@task(cache=True, cache_version="0.1")
def create_model() -> nn.Module:
return nn.Linear(1, 1)
@workflow
def wf(param: int = 1):
model = create_model()
sub_wf(model=model, param=param)
Screenshots
The first execution with param=1
results in cache puts for all tasks:
Screenshot 2023-07-06 at 15 57 48▾
Screenshot 2023-07-06 at 16 09 30▾
other_task
has a cache miss since we changed param
. However, since create_model
had a cache hit, so should have train
.
Instead, one can observe that the output of create_model
(retrieved from cache) ...
Screenshot 2023-07-06 at 16 09 44▾
Screenshot 2023-07-06 at 16 09 55▾