#4 Bump grpcio from 1.51.3 to 1.53.0 in /scipy-2023 Pull request opened by dependabot[bot] ⚠️ Dependabot is rebasing this PR ⚠️ Rebasing might not happen immediately, so don't worry if this takes some time. Note: if you make any changes to this PR yourself, they will take precedence over the rebase. * * * Bumps grpcio from 1.51.3 to 1.53.0. Release notes Sourced from grpcio's releases.

Release v1.53.0

This is release 1.53.0 (glockenspiel) of gRPC Core.

For gRPC documentation, see grpc.io. For previous releases, see Releases.

This release contains refinements, improvements, and bug fixes, with highlights listed below.

Core

• xDS: fix crash when removing the last endpoint from the last locality in weighted_target. (#32592)

• filter stack: pass peer name up via recv_initial_metadata batch. (#31933)

• [EventEngine] Add advice against blocking work in callbacks. (#32397)

• [http2] Dont drop connections on metadata limit exceeded. (#32309)

• xDS: reject aggregate cluster with empty cluster list. (#32238)

• Fix Python epoll1 Fork Support. (#32196)

• server: introduce ServerMetricRecorder API and move per-call reporting from a C++ interceptor to a C-core filter. (#32106)

• [EventEngine] Add invalid handle types to the public API. (#32202)

• [EventEngine] Refactoring the EventEngine Test Suite: Part 1. (#32127)

• xDS: fix WeightedClusters total weight handling. (#32134)

C++

• Update minimum MSVC version to 2019. (#32615)

• Use CMake variables for paths in pkg-config files. (#31671)

C#

• Grpc.Tools: Use x86 protoc binaries on arm64 Windows. (#32017)

Python

• Support python 3.11 on aarch64. (#32270)

• Include .pyi file. (#32268)

• De-experimentalize wait-for-ready. (#32143)

• De-experimentalize compression. (#32138)

Ruby

• [ruby]: add pre-compiled binaries for ruby 3.2; drop them for ruby 2.6. (#32089)

Release v1.53.0-pre2

This is a prerelease of gRPC Core 1.53.0 (glockenspiel).

For gRPC documentation, see grpc.io. For previous releases, see Releases.

... (truncated) Commits • `358bfb5` Bump version to 1.53.0 (#32685) • `6e1ebe7` Backport: Ensure compatibility with the new custom kokoro win2019 image (#326... • `44a77f6` Backport 1.53: Update minimum MSVC version to 2019 (#32615) • `c11153c` backport to 1.53: xDS: fix crash when removing the last endpoint from the las... • `7c7712a` Bump version to 1.53.0-pre2. (#32545) • `a4017dc` backport to 1.53: [promises] Make Poll<T> its own type, not a variant<> (#32540) • `3f93c16` Fuzzer fix backport to v1.53 (#32511) • `5b244b2` Bump release version to 1.53.0-pre1 (#32428) • `6589340` Bump core version 202302161703 (#32416) • `d49e151` [backoff] Add random early detection classifier (#32354) • Additional commits viewable in compare view Dependabot compatibility score Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting

@dependabot rebase

. * * * Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: •

@dependabot rebase

will rebase this PR •

@dependabot recreate

will recreate this PR, overwriting any edits that have been made to it •

@dependabot merge

will merge this PR after your CI passes on it •

@dependabot squash and merge

will squash and merge this PR after your CI passes on it •

@dependabot cancel merge

will cancel a previously requested merge and block automerging •

@dependabot reopen

will reopen this PR if it is closed •

@dependabot close

will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually •

@dependabot ignore this major version

will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) •

@dependabot ignore this minor version

will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) •

@dependabot ignore this dependency

will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the Security Alerts page. flyteorg/flyte-conference-talks GitHub Actions: build (integration, scipy-2023, 3.9) GitHub Actions: build (unit, scipy-2023, 3.9) ✅ 2 other checks have passed 2/4 successful checks

#1722 Add a couple tests Pull request opened by wild-endeavor TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit GitHub Actions: build-plugins (3.8, flytekit-onnx-pytorch) GitHub Actions: build-plugins (3.8, flytekit-modin) GitHub Actions: build-plugins (3.8, flytekit-mlflow) GitHub Actions: build-plugins (3.8, flytekit-kf-tensorflow) GitHub Actions: build-plugins (3.8, flytekit-kf-pytorch) GitHub Actions: build-plugins (3.8, flytekit-kf-mpi) GitHub Actions: build-plugins (3.8, flytekit-k8s-pod) GitHub Actions: build-plugins (3.8, flytekit-huggingface) GitHub Actions: build-plugins (3.8, flytekit-hive) GitHub Actions: build-plugins (3.8, flytekit-greatexpectations) GitHub Actions: build-plugins (3.8, flytekit-envd) GitHub Actions: build-plugins (3.8, flytekit-duckdb) GitHub Actions: build-plugins (3.8, flytekit-dolt) GitHub Actions: build-plugins (3.8, flytekit-deck-standard) GitHub Actions: build-plugins (3.8, flytekit-dbt) GitHub Actions: build-plugins (3.8, flytekit-data-fsspec) GitHub Actions: build-plugins (3.8, flytekit-dask) GitHub Actions: build-plugins (3.8, flytekit-bigquery) GitHub Actions: build-plugins (3.8, flytekit-aws-sagemaker) GitHub Actions: build-plugins (3.8, flytekit-aws-batch) GitHub Actions: build-plugins (3.8, flytekit-aws-athena) GitHub Actions: lint GitHub Actions: build (windows-latest, 3.11) GitHub Actions: build (windows-latest, 3.9) GitHub Actions: build (windows-latest, 3.8) GitHub Actions: build (ubuntu-latest, 3.11) GitHub Actions: build (ubuntu-latest, 3.10) GitHub Actions: build (ubuntu-latest, 3.9) GitHub Actions: build (ubuntu-latest, 3.8) ✅ 1 other check has passed 1/30 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytepropeller/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flytepropeller/commit/206013ab02af16d326cea394807a1743eefb82ec|206013ab>

- Fix usage of archive replacements (#586) flyteorg/flytepropeller

#412 Remove flytekit image priming Pull request opened by jeevb Smoke tested on

demo

and

sandbox

. flyteorg/flytectl GitHub Actions: Goreleaser GitHub Actions: Generate documentation GitHub Actions: Bump Version GitHub Actions: Dry Run Goreleaser ✅ 8 other checks have passed 8/12 successful checks

Release - v1.1.103 New release published by flyte-bot Changelog • 206013a Fix usage of archive replacements (#586) flyteorg/flytepropeller

1 new commit pushed to

<https://github.com/flyteorg/flytectl/tree/master|master>

by jeevb

<https://github.com/flyteorg/flytectl/commit/ac3aa739aa7a499179a298715641aaf9ca836193|ac3aa739>

- Remove flytekit image priming (#412) flyteorg/flytectl

#3837 Update auth config instructions Pull request opened by davidmirror-ops Tracking issue Closes #3029 Describe your changes A first attempt to provide updated and easier to follow instructions to setup auth for both

flyte-binary

and

flyte-core

I haven't been able to reproduce the Azure AD config as external auth server, so it is left as a separate section to be addressed in an upcoming PR Check all the applicable boxes ☑︎ I updated the documentation accordingly. ☑︎ All new and existing tests passed. ☑︎ All commits are signed-off. Screenshots Note to reviewers flyteorg/flyte ✅ All checks have passed 6/6 successful checks

Release - v0.6.40 New release published by flyte-bot Changelog • ac3aa73 Remove flytekit image priming (#412) flyteorg/flytectl

1 new commit pushed to

<https://github.com/flyteorg/homebrew-tap/tree/main|main>

by flyte-bot

<https://github.com/flyteorg/homebrew-tap/commit/1938dd8099ea7cb2da6b6b46a47a80b07ec7e57a|1938dd80>

- Brew formula update for flytectl version v0.6.40 flyteorg/homebrew-tap

1 new commit pushed to

<https://github.com/flyteorg/flyteadmin/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flyteadmin/commit/53282fe979b628447e72d0e3f8fd0e2b9235d929|53282fe9>

- remove welcome bot from boilerplate (#460) flyteorg/flyteadmin

#460 remove welcome bot from boilerplate Pull request opened by samhita-alla Signed-off-by: Samhita Alla aallasamhita@gmail.com TL;DR This PR removes welcome bot config from boilerplate cause it's moved to .github repository. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ flyteorg/flyte#1963 Follow-up issue flyteorg/flyteadmin GitHub Actions: Build & Push Flyteadmin Image GitHub Actions: Build & Push Flytescheduler Image GitHub Actions: Goreleaser GitHub Actions: Bump Version Codecov: 58.58% (-0.04%) compared to 78b3e14 ✅ 9 other checks have passed 9/14 successful checks

#1015 more formatting cleanup Pull request opened by cosmicBboy flyteorg/flytesnacks GitHub Actions: Build & Push Example Image to GHCR (examples/k8s_pod_plugin) GitHub Actions: Build & Push Example Image to GHCR (examples/k8s_dask_plugin) GitHub Actions: Build & Push Example Image to GHCR (examples/image_spec) GitHub Actions: Build & Push Example Image to GHCR (examples/house_price_prediction) GitHub Actions: Build & Push Example Image to GHCR (examples/greatexpectations_plugin) GitHub Actions: Build & Push Example Image to GHCR (examples/forecasting_sales) GitHub Actions: Build & Push Example Image to GHCR (examples/feast_integration) GitHub Actions: Build & Push Example Image to GHCR (examples/exploratory_data_analysis) GitHub Actions: Build & Push Example Image to GHCR (examples/duckdb_plugin) GitHub Actions: Build & Push Example Image to GHCR (examples/dolt_plugin) GitHub Actions: Build & Push Example Image to GHCR (examples/dbt_plugin) GitHub Actions: Build & Push Example Image to GHCR (examples/databricks_plugin) GitHub Actions: Build & Push Example Image to GHCR (examples/control_flow) GitHub Actions: Build & Push Example Image to GHCR (examples/containerization) GitHub Actions: Build & Push Example Image to GHCR (examples/blast) GitHub Actions: Build & Push Example Image to GHCR (examples/bigquery_plugin) GitHub Actions: Build & Push Example Image to GHCR (examples/basics) GitHub Actions: Build & Push Example Image to GHCR (examples/aws_batch_plugin) GitHub Actions: Build & Push Example Image to GHCR (examples/athena_plugin) GitHub Actions: lint (3.11) GitHub Actions: lint (3.10) GitHub Actions: lint (3.9) GitHub Actions: lint (3.8) GitHub Actions: Docs Warnings ✅ 3 other checks have passed 3/27 successful checks

#586 Fix archive.replacements in goreleaser Pull request opened by eapolinario TL;DR Similar to flyteorg/flytepropeller#586 Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Same fix as flyteorg/flytepropeller#586. Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flyteadmin GitHub Actions: Integration Test / Integration tests GitHub Actions: End2End Test / End to End tests ✅ 8 other checks have passed 8/10 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>

by cosmicBboy

<https://github.com/flyteorg/flyte-conference-talks/commit/96233a12dde770be5834cb6fc1a4ab8434da6fbe|96233a12>

- update scipy-2023 readme, workflows, tests flyteorg/flyte-conference-talks

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by samhita-alla

<https://github.com/flyteorg/flyte/commit/58a1de6dd4df4765aeccce2008bb69eeee1f87ec|58a1de6d>

- Fix broken links (#3830) flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytesnacks/tree/master|master>

by pingsutw

<https://github.com/flyteorg/flytesnacks/commit/73840e67cf58ee6c3372c0afb8f15c9a63c54771|73840e67>

- Update Athena example (#1007) flyteorg/flytesnacks

#1723 Add tags to execution Pull request opened by pingsutw TL;DR Blocked by flyteorg/flyteadmin#571

pyflyte run --remote --tags '["hello", "flyte"]' workflow.py wf

Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Added a new arg (tags) to execution Tracking Issue flyteorg/flyte#3320 Follow-up issue NA flyteorg/flytekit GitHub Actions: build-plugins (3.11, flytekit-sqlalchemy) GitHub Actions: build-plugins (3.11, flytekit-snowflake) GitHub Actions: build-plugins (3.11, flytekit-polars) GitHub Actions: build-plugins (3.11, flytekit-papermill) GitHub Actions: build-plugins (3.11, flytekit-pandera) GitHub Actions: build-plugins (3.11, flytekit-kf-tensorflow) GitHub Actions: build-plugins (3.11, flytekit-kf-pytorch) GitHub Actions: build-plugins (3.11, flytekit-hive) GitHub Actions: build-plugins (3.11, flytekit-envd) ✅ 21 other checks have passed 21/30 successful checks

#3839 [BUG] launching a registered workflow with a `Union` type breaks the WebUI Issue created by alekpikl Describe the bug Minimal reproducible example:

from typing import Union

from flytekit import task, workflow

@task()
def my_task(number: Union[int, str]):
    print(number)


@workflow
def my_work(number: Union[int, str] = 1):
    my_task(number=number)

Registering the workflow works:

pylyte register workflow.py

✅. Running the workflow directly works:

pylyte run --remote run workflow.py my_workflow

✅. Trying to run the registered workflow from the WebUI fails ❌ (see screenshot). Expected behavior I expect an error at registration or in the WebUI if the workflow is broken. Currently, the WebUI errors out without a hint. Additional context to reproduce No response Screenshots The error in the WebUI: More information by pressing Fn12: Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

Release - flytesnacks v0.3.210 New release published by flyte-bot Changelog • 73840e6 Update Athena example (#1007) • a49ba51 fix formatting, add references (#1014) flyteorg/flytesnacks

#236 Align java examples with scala ones Pull request opened by honnix TL;DR Align java examples with scala ones, for the subworkflow related things. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☐ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description It is a bit confusing that

SubWorkflow

in Java and Scala carries different meaning for different purpose. So we align the Java one with Scala. Tracking Issue NA Follow-up issue NA flyteorg/flytekit-java ✅ All checks have passed 3/3 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by honnix

<https://github.com/flyteorg/flytekit-java/commit/54a94b48e52b1a09788eaf527fcb1166dfb8506e|54a94b48>

- Align java examples with scala ones (#236) flyteorg/flytekit-java

#1016 Add dockerfile to snowflake example Pull request opened by pingsutw Failed to generate snowflake-plugin.tar.gz here, because snowflake example doesn't have dockerfile. We use this script to generate a tar file, and it requires every example to have a dockerfile flyteorg/flytesnacks GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (examples/whylogs_plugin) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (examples/type_system) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (examples/sagemaker_pytorch_plugin) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (examples/nlp_processing) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (examples/kfmpi_plugin) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (examples/k8s_spark_plugin) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (examples/forecasting_sales) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (examples/dbt_plugin) ✅ 22 other checks have passed 22/30 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytesnacks/tree/master|master>

by cosmicBboy

<https://github.com/flyteorg/flytesnacks/commit/80427a51dda0edd78c931077ca69213c444887aa|80427a51>

- more formatting cleanup (#1015) flyteorg/flytesnacks

#3840 [BUG] Snowflake plugin configuration should not take short-lived JWT token Issue created by iaroslav-ciupin Describe the bug Synopsis Snowflake plugin integration should not require user to store JWT token in propeller secret. JWT token is short-lived and is valid for only 10 minutes(or so), which means user will have to re-generate new token and update propeller secret every 10 minutes. This is a very bad UX. Instead , there should be a different way to configure Snowflake such that plugin generates JWTs. Expected behavior User should be able to configure Snowflake integration once and it should work without any future adjustments. Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytepropeller/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flytepropeller/commit/0cfb0a0ad73ca97036311f4d18d908260bfa601c|0cfb0a0a>

- Added IsFailurePermanent flag on DynamicTaskStatus (#567) flyteorg/flytepropeller

#3606 [BUG] Runtime error causes retries Issue created by gregfriedland Describe the bug When a task fails with a user-generated

RuntimeError

, the workflow is retried. According to the docs, it seems like this case is considered non-recoverable and should not be retried. Expected behavior Workflow and task fail with no retries Additional context to reproduce No response Screenshots Traceback (most recent call last):

File "/app/pip_pypi__flytekit/flytekit/exceptions/scopes.py", line 203, in user_entry_point
    return wrapped(*args, **kwargs)
  File "/app/atomwise/...", line 266, in run_block
    block.validate_output(cmpds, result)
  File "/app/atomwise/..., line 119, in validate_output
    raise RuntimeError(

Message:

<blah>

User error. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#567 Added IsFailurePermanent flag on DynamicTaskStatus Pull request opened by hamersaw TL;DR Currently dynamic tasks treat all failures and retryable which results in unintended behavior when subtasks are attempted multiple times. This PR adds a

IsFailurePermanent

field on the

DynamicTaskStatus

struct to indicate that the failure is permanent, and therefore should not be recovered. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description The choice to use a new flag is entirely out of backwards compatibility. Other options include: (1) Using the ExecutionError in flyteplugins io rather than flyteidl ExecutionError: This would make retrieval of in progress dynamic task status fail when it expects a different error type on the field. (2) Naming the flag

IsRecoverable

vs `isPermanent`: This is difficult because intuition says that we should use

IsRecoverable

which adheres more to the Flyte standard. The biggest issue is that dynamics treat all failures as retryable by default now, so this would change the default to be non-recoverable (ie.

false

for

IsRecoverable

) which changes the default behavior. It is VERY difficult to ensure there will be no regressions. Tracking Issue fixes flyteorg/flyte#3606 Follow-up issue NA flyteorg/flytepropeller GitHub Actions: Build & Push Flytepropeller Image GitHub Actions: Goreleaser GitHub Actions: Bump Version ✅ 11 other checks have passed 11/14 successful checks

#3841 [BUG] Task templates are broken by deck upload logic Issue created by cosmicBboy Describe the bug flyteorg/flytekit#1693 breaks task templates like

SQLite3Task

. For example running the following task:

QUERY = "SELECT species, bill_length_mm, bill_depth_mm, flipper_length_mm, body_mass_g FROM penguins"
get_data = SQLite3Task(
    name="get_penguins_data",
    query_template=QUERY,
    output_schema_type=PenquinsDataset,
    task_config=SQLite3Config(
        uri="<https://datasette-seaborn-demo.datasette.io/penguins.db>"
    ),
)

Produces an error:

[1/1] currentAttempt done. Last Error: USER::�                                                                              │
│ /usr/local/lib/python3.9/site-packages/flytekit/exceptions/scopes.py:160 in  │
│ system_entry_point                                                           │
│                                                                              │
│ ❱ 160 │   │   │   │   return wrapped(*args, **kwargs)                        │
│                                                                              │
│ /usr/local/lib/python3.9/site-packages/flytekit/bin/entrypoint.py:362 in     │
│ _execute_task                                                                │
│                                                                              │
│ ❱ 362 │   │   _handle_annotated_task(ctx, _task_def, inputs, output_prefix)  │
│                                                                              │
│ /usr/local/lib/python3.9/site-packages/flytekit/bin/entrypoint.py:304 in     │
│ _handle_annotated_task                                                       │
│                                                                              │
│ ❱ 304 │   _dispatch_execute(ctx, task_def, inputs, output_prefix)            │
│                                                                              │
│ /usr/local/lib/python3.9/site-packages/flytekit/bin/entrypoint.py:163 in     │
│ _dispatch_execute                                                            │
│                                                                              │
│ ❱ 163 │   if not task_def.disable_deck:                                      │
╰──────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'ExecutableTemplateShimTask' object has no attribute 
'disable_deck'

Expected behavior The

ExecutableTemplateShimTask

class doesn't have a

disable_deck

attribute. Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1724 use getattr to access task_def.disable_deck in entrypoint Pull request opened by cosmicBboy Fixes flyteorg/flyte#3841 TL;DR #1693 introduced a bug in task templates like

SQlite3Task

. This PR fixes an issue in the execution entrypoint that assumes the all task objects have the

disable_deck

attribute. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Use the

getattr

function to get

disable_deck

, and return

False

if not present. Tracking Issue https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit GitHub Actions: build-plugins (3.8, flytekit-huggingface) GitHub Actions: build-plugins (3.8, flytekit-hive) GitHub Actions: build-plugins (3.8, flytekit-greatexpectations) GitHub Actions: build-plugins (3.8, flytekit-envd) GitHub Actions: build-plugins (3.8, flytekit-duckdb) GitHub Actions: build-plugins (3.8, flytekit-dolt) GitHub Actions: build-plugins (3.8, flytekit-deck-standard) GitHub Actions: build-plugins (3.8, flytekit-dbt) GitHub Actions: build-plugins (3.8, flytekit-data-fsspec) GitHub Actions: build-plugins (3.8, flytekit-dask) GitHub Actions: build-plugins (3.8, flytekit-bigquery) GitHub Actions: build-plugins (3.8, flytekit-aws-sagemaker) GitHub Actions: build-plugins (3.8, flytekit-aws-batch) GitHub Actions: build-plugins (3.8, flytekit-aws-athena) GitHub Actions: lint GitHub Actions: build (windows-latest, 3.11) GitHub Actions: build (windows-latest, 3.9) GitHub Actions: build (windows-latest, 3.8) GitHub Actions: build (ubuntu-latest, 3.11) GitHub Actions: build (ubuntu-latest, 3.10) GitHub Actions: build (ubuntu-latest, 3.9) GitHub Actions: build (ubuntu-latest, 3.8) GitHub Actions: docs GitHub Actions: Docs Warnings ✅ 2 other checks have passed 2/26 successful checks

#3842 [BUG] Dynamic workflows break caching of artifacts offloaded to blob storage Issue created by fg91 Describe the bug Some type transformers offload artifacts to blob storage as follows (e.g. here):

def to_literal(...) -> Literal:
        local_path = ...
        # Save object to local path

        remote_path = ctx.file_access.get_random_remote_path(local_path)
        ctx.file_access.put_data(local_path, remote_path, is_multipart=False)
        # Return Literal containing remote_path

When objects of such types are passed to

@dynamic

workflows and then passed along to tasks called within the dynamic workflow, this behaviour always leads to cache misses. The reason is that in the dynamic workflow, the objects are deserialized and then again serialized to a different random remote path. Expected behavior There should not be cache misses in this situation. Additional context to reproduce Let us consider this example workflow:

import torch.nn as nn
from flytekit import task, workflow, dynamic


@task(cache=True, cache_version="0.1")
def train(model: nn.Module) -> nn.Module:
    print(f"Training model {model}")
    return model


@task(cache=True, cache_version="0.1")
def other_task(param: int) -> int:
    print(f"Doing something else with param {param}")
    return param


@dynamic(cache=True, cache_version="0.1")
def sub_wf(model: nn.Module, param: int) -> tuple[nn.Module, int]:
    other_task(param=param)
    train(model=model)

    return model, param


@task(cache=True, cache_version="0.1")
def create_model() -> nn.Module:
    return nn.Linear(1, 1)


@workflow
def wf(param: int = 1):
    model = create_model()
    sub_wf(model=model, param=param)

Screenshots The first execution with

param=1

results in cache puts for all tasks: Next, let's re-run the workflow but with `param=2`: It is expected that

other_task

has a cache miss since we changed

param

. However, since

create_model

had a cache hit, so should have

train

. Instead, one can observe that the output of

create_model

(retrieved from cache) ... ... is not the same is as the input to `train`: The reason is that the respective type transformer deserialized the artifact and serialized it again to a new random bucket path. I don't know how exactly this could be solved but the type engine should "somehow detect that it is being executed in a dynamic workflow" and should "simply forward the artifacts instead of serializing them to a different location". Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte