#1582 Feature/3506: Improve Type Conversion errors, use rich to prettify error messages Pull request opened by cosmicBboy TL;DR This PR makes input/output type conversion error messages a lot more useful by providing more context to the user. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added • [ ] Code documentation added • [ ] Any pending items have an associated Issue Complete description This PR makes input/output type conversion error messages a lot more actionable by: • providing context on what workflow and task was executing when the error was raised • providing context on the input name / output name and positional index that raised the error • modifying the string representation of flyte literal types to be more explicit • pretty-printing errors using

rich

Tracking Issue flyteorg/flyte#3506 Follow-up issue NA flyteorg/flytekit ✅ All checks have passed 2/2 successful checks

#4 Add LinkedIn to list of adopters Pull request opened by goyalankit flyteorg/community ✅ All checks have passed 2/2 successful checks

1 new commit pushed to

<https://github.com/flyteorg/community/tree/main|main>

by kumare3

<https://github.com/flyteorg/community/commit/f8c06448561a27b136c6a02faaa619d4d5fd35f1|f8c06448>

- Update ADOPTERS.md (#4) flyteorg/community

#1576 Feat: Add torchrun plugin Pull request opened by fg91 on <!date^1680530978^{date_short}|2023-04-03T14:09:38Z> TL;DR Work in progress This plugin allows running torch elastic (torchrun) distributed training with Flyte.

from dataclasses import dataclass

import torch
from dataclasses_json import dataclass_json
from flytekit import dynamic, task, workflow
from flytekitplugins.kfpytorch import PyTorch

from .torch_elastic_task import Elastic


@dataclass_json
@dataclass
class Config:
    lr: float = 1e-5
    bs: int = 64
    name: str = "foo"


@task
def init_model() -> torch.nn.Module:
    model = torch.nn.Linear(11, 22)

    return model


"""
This doesn't start a kubelfow pytorch job yet but a single python task Pod which then
runs a local worker group in sub-processes.
The changes in the flyteidl protobuf definitions, the flytekit python api, and the
flytepropeller (operator) which we need to actually make this distributed on multiple nodes
are easy (see RFC document linked in PR description).
"""
@task(
    task_config=Elastic(
        min_replicas=1,
        max_replicas=1,
        start_method="spawn",
    )
)
def train(config: Config, model: torch.nn.Module) -> tuple[str, Config, torch.nn.Module]:
    import os

    import torch

    local_rank = os.environ["LOCAL_RANK"]

    out_model = torch.nn.Linear(1000, int(local_rank) * 2000 + 1)
    print(f"Training with config {config}")
    config.name = "modified"
    return f"result from local rank {local_rank}", config, out_model


@workflow
def wf(config: Config=Config()) -> tuple[str, Config, torch.nn.Module]:
    model = init_model()
    return train(config=config, model=model)


if __name__ == "__main__":
    print(wf(config=Config()))

Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

#1129 Add log streaming to papermill plugin Pull request opened by CalvinLeather TL;DR Enabling the Meant to open this in my fork for internal review, but will leave up as draft and wrap up tomorrow with a better test + smoke test. Smoke test: Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

#3053 [Core feature] control ENV var name for injected secrets Issue created by flixr Motivation: Why do you think this is important? Split out from flyteorg/flytekit#1307: I need to inject k8s secrets as env vars into my raw ContainerTasks (and actually for PythonTasks as well). As an example I want to inject

AWS_ACCESS_KEY_ID

and

AWS_SECRET_ACCESS_KEY

. Essentially I want to do what k8s already nicely provides: https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#define-a-container-environment-variable-with-data-from-a-single-secret Using flyte secrets there is no way to control the ENV var name, something like e.g.

Secret(group="minio-write-creds", key="AWS_ACCESS_KEY_ID", mount_requirement=Secret.MountType.ENV_VAR)

will inject the secret as

_FSEC_MINIO_WRITE_CREDS_AWS_ACCESS_KEY_ID

and this is currently hardcoded in flytepropeller. This seems sensible iff you need to get the secret in Python code via flytekit context/secret manager, but not for raw containers. Even for a Python task it would often be easier to just control the env var name (as e.g. in the case of the S3 credentials that boto3 would automatically pick up then). Goal: What should the final outcome look like, ideally? Maybe something like

Secret(group="minio-write-creds", key="access_key", mount_requirement=Secret.MountType.ENV_VAR, name="AWS_ACCESS_KEY_ID")

To decouple group/key name from env var name, just like k8s itself. If name is not given it could still work by naming as prefix + group + key. Describe alternatives you've considered Working around this by wrapping my container task in a shell script that pull out the secret from the env var (or file) and populate the correct env var for the actual process. But this is super cumbersome and actually requires the wrapper script to know about the group name (so k8s secret name), which makes it super awkward to change credentials by just using a different secret. Same goes for secrets injected as file... Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3574 [BUG] unset CPU limit is overridden with CPU request Issue created by flixr Describe the bug A container task with only CPU requests but no limits set, still gets limits applied:

from flytekit import ContainerTask, kwtypes, workflow, Resources

hello_task = ContainerTask(
    name="hello",
    image="ubuntu:20.04",
    requests=Resources(cpu="2", mem="1Gi"),
    limits=Resources(mem="2Gi"),
    command=["echo", "hello"]
)

Running this task results in a pod with cpu limit also set to 2 (same as request), but there should be no cpu limit. Expected behavior If no CPU limit is specified, it is also not set implicitly and no CPU limit is applied in k8s. So I propose to only copy the requests to limits as a whole if limits are completely unset instead of filling missing limits with the respective requests values. Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3575 [UI Feature] Ability to inspect details of all Launch Plan version Issue created by pradithya Motivation: Why do you think this is important? The launch plan details page is able to show only the latest launch plan version. Clicking the list of launch plan versions will lead to the same version list but with more entries. It's not ideal especially if we have an active scheduled launch plan which is not the latest version. The only way to access inspect the property of such launch plan is via flytectl. Goal: What should the final outcome look like, ideally? Similar UX such as the workflow version selection table, but instead of workflow graph it should show the relevant launch plan version property. Additionally, it's also useful if the launch plan detailed page shows the active launch plan by default. If no active launch plan then it should fall back to show the latest version. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by wild-endeavor

<https://github.com/flyteorg/flyte/commit/bc521dac6f0dc6e1942efa6447611cd3354540c4|bc521dac>

- Update Flyte components (#3572) flyteorg/flyte

Release - Flyte v1.5.0 milestone release New release published by flyte-bot Flyte 1.5 release Platform We're laying the foundation for an improved experience to help performance investigations. Stay tuned for more details! We can now submit Ray jobs to separate clusters (other than the one flytepropeller is running). Thanks to flyteorg/flyteplugins#321. Several bug fixes, including: • Fix fast-cache bug on first node event • Split flyte-binary services into http and grpc in helm charts Database Migrations One of the improvements planned requires us to clean up our database migrations. We have done so in this release so you should see a series of new migrations. These should have zero impact if you are otherwise up-to-date on migrations (which is why they are all labeled

noop

) but please be aware that it will add a minute or so to the init container/command that runs the migrations in the default Helm charts. Notably, because these should be a no-op, they also do not come with any rollback commands. If you experience any issues, please let us know. Flytekit Python 3.11 is now officially supported. Revamped Data subsystem The data persistence layer was completely revamped. We now rely exclusively on fsspec to handle IO. Most users will benefit from a more performant IO subsystem, in other words, no change is needed in user code. The data persistence layer has undergone a thorough overhaul. We now exclusively utilize fsspec for managing input and output operations. For the majority of users, the improved IO subsystem provides enhanced performance, meaning that no modifications are required in their existing code. This change opened the door for flytekit to rely on fsspec streaming capabilities. For example, let's say we want to stream a file, now we're able to do:

@task
def copy_file(ff: FlyteFile) -> FlyteFile:
    new_file = FlyteFile.new_remote_file(ff.remote_path)
    with ff.open("r", cache_type="simplecache", cache_options={}) as r:
        with new_file.open("w") as w:
            w.write(r.read())
    return new_file

This feature is marked as experimental. We'd love feedback on the API! Limited support for partial tasks We can use functools.partial to "freeze" some task arguments. Let's take a look at an example where we partially fix the parameter for a task:

@task
def t1(a: int, b: str) -> str:
    return f"{a} -> {b}"
    
t1_fixed_b = functools.partial(t1, b="hello")

@workflow
def wf(a: int) -> str:
    return t1_fixed_b(a=a)

Notice how calls to

t1_fixed_b

do not need to specify the

b

parameter. This also works for MapTasks in a limited capacity. For example:

from flytekit import task, workflow, partial, map_task

@task
def t1(x: int, y: float) -> float:
    return x + y

@workflow
def wf(y: List[float]):
   partial_t1 = partial(t1, x=5)
   return map_task(partial_t1)(y=y)

We are currently seeking feedback on this feature, and as a result, it is labeled as experimental for now. Also worth mentioning that fixing parameters of type list is not currently supported. For example, if we try to register this workflow:

from functools import partial
from typing import List
from flytekit import task, workflow, map_task

@task
def t(a: int, xs: List[int]) -> str:
    return f"{a} {xs}"

@workflow
def wf():
    partial_t = partial(t, xs=[1, 2, 3])
    map_task(partial_t)(a=[1, 2])

We're going to see this error:

❯ pyflyte run workflows/example.py wf
Failed with Unknown Exception <class 'ValueError'> Reason: Map tasks do not support partial tasks with lists as inputs.
Map tasks do not support partial tasks with lists as inputs.

Flyteconsole Multiple bug fixes around waiting for external inputs. Better support for dataclasses in the launch form. flyteorg/flyte

Deployment to github-pages by flyte-bot flyteorg/flyte

#385 Flyte address resolution Pull request opened by wild-endeavor TL;DR Add a quick lookup for task input/output Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flyteidl Codecov: 76.11% (-0.61%) compared to bdb7630 ✅ 12 other checks have passed 12/13 successful checks

Release - v1.5.0 - April 2023 New release published by wild-endeavor Data subsystem now uses fsspec, partial support for maps, python 3.11, flyte file/directory streaming, device auth flow. Please see the Flyte release notes for usage snippets. Full changelog below. What's Changed • prefer FLYTE_ prefixed AWS creds env vars by @flixr in #1523 • Filter out remote entity when generating pb by @honnix in #1545 • Pyflyte docs formatting by @bryanwweber in #1538 • Data subsystem by @wild-endeavor in #1526 • Less strict docker dependency versions by @jcugat in #1536 • Pluck retry from flytekit and into sagemaker by @eapolinario in #1411 • Update the pypi wait by @wild-endeavor in #1554 • Stream Directories and Files using Flyte by @kumare3 in #1512 • Make

FlyteFile

compatible with

Annotated[..., HashMethod]

by @AdrianoKF in #1544 • move FlyteSchema deprecation warning to initialization method by @cosmicBboy in #1558 • add pod_template and pod_template_name arguments for ContainerTask by @flixr in #1515 • Pass locally defined scopes to RemoteClientConfigStore by @franco-bocci in #1553 • TypeTransformer for TensorFlow model by @samhita-alla in #1562 • Remove flytekit-fsspec plugin from default dockerfile by @wild-endeavor in #1561 • Device auth flow / Headless auth by @kumare3 in #1552 • support python 3.11 by @cosmicBboy in #1557 • url encode secret in client credentials flow by @wild-endeavor in #1566 • Python run multiple files by @pingsutw in #1559 • General Partial support in flytekit and multi-list support in flytekit by @kumare3 in #1556 • fix: Silence keyring warnings by changing to debug by @ggydush in #1568 • Support GCP secrets by @wild-endeavor in #1571 • Automatically remove unused import by @pingsutw in #1574 • Disallow partial lists in map tasks by @eapolinario in #1577 • Remove duplicate reporting logic by @wild-endeavor in #1578 • [Core feature] Convert List[Any] to a single pickle file by @Yicheng-Lu-llll in #1535 • Improve authoring structure documentation by @samhita-alla in #1572 New Contributors • @bryanwweber made their first contribution in #1538 • @franco-bocci made their first contribution in #1553 • @ggydush made their first contribution in #1568 • @Yicheng-Lu-llll made their first contribution in #1535 Full Changelog: v1.4.2...v1.5.0 flyteorg/flytekit

#56 Lazily initialize kubernetes client only when using kube api watcher Pull request opened by jeevb TL;DR Lazily load k8s client only in kubeapi watcher. This prevents init container errors when running with service accounts that have

automountServiceAccountToken: false

set. Smoke tested in sandbox with and without

automountServiceAccountToken: false

, with both the default shared process namespace watcher and kubeapi watcher respectively. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Moved initialization of k8s client from copilot root command to

NewKubeAPIWatcher

function. Now it is only initialized when the

kube-api

watcher is specified. flyteorg/flytecopilot ✅ All checks have passed 2/2 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytecopilot/tree/master|master>

by jeevb

<https://github.com/flyteorg/flytecopilot/commit/12f9ff5370a929f2f2b986adbf203f90766dd50c|12f9ff53>

- Lazily initialize kubernetes client only when using kube api watcher (#56) flyteorg/flytecopilot

Release - v0.0.29 New release published by flyte-bot Changelog • 12f9ff5 Lazily initialize kubernetes client only when using kube api watcher (#56) flyteorg/flytecopilot

#392 Rust pb Pull request opened by eapolinario Read then delete this section - Make sure to use a concise title for the pull-request. _- Use #patch, #minor or #major in the pull-request title to bump the corresponding version. Otherwise, the patch version will be bumped. More details_ TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flyteidl ✅ All checks have passed 13/13 successful checks

#1584 Dockerfile for sqlalchemy for 3.11 Pull request opened by wild-endeavor TL;DR Just add a new dockerfile for now, holding up CI. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin https://github.com/flyteorg/flytekit/actions/runs/4631841063 flyteorg/flytekit GitHub Actions: build-plugins (3.8, flytekit-modin) GitHub Actions: build-plugins (3.8, flytekit-mlflow) GitHub Actions: build-plugins (3.8, flytekit-kf-tensorflow) GitHub Actions: build-plugins (3.8, flytekit-kf-pytorch) GitHub Actions: build-plugins (3.8, flytekit-kf-mpi) GitHub Actions: build-plugins (3.8, flytekit-k8s-pod) GitHub Actions: build-plugins (3.8, flytekit-hive) GitHub Actions: build-plugins (3.8, flytekit-greatexpectations) GitHub Actions: build-plugins (3.8, flytekit-duckdb) GitHub Actions: build-plugins (3.8, flytekit-dolt) GitHub Actions: build-plugins (3.8, flytekit-deck-standard) GitHub Actions: build-plugins (3.8, flytekit-dbt) GitHub Actions: build-plugins (3.8, flytekit-data-fsspec) GitHub Actions: build-plugins (3.8, flytekit-dask) GitHub Actions: build-plugins (3.8, flytekit-bigquery) GitHub Actions: build-plugins (3.8, flytekit-aws-sagemaker) GitHub Actions: build-plugins (3.8, flytekit-aws-batch) GitHub Actions: build-plugins (3.8, flytekit-aws-athena) GitHub Actions: lint GitHub Actions: build (windows-latest, 3.11) GitHub Actions: build (windows-latest, 3.9) GitHub Actions: build (windows-latest, 3.8) GitHub Actions: build (ubuntu-latest, 3.11) GitHub Actions: build (ubuntu-latest, 3.10) GitHub Actions: build (ubuntu-latest, 3.9) GitHub Actions: build (ubuntu-latest, 3.8) GitHub Actions: docs GitHub Actions: Docs Warnings ✅ 2 other checks have passed 2/30 successful checks

#3577 [Housekeeping] Add a RequestID field to contextutils and logger Issue created by EngHabu Describe the issue For scenarios when we want to augment context with a request id and have it logged using the underlying logger. What if we do not do this? Overload one of the existing logging fields Related component(s) No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#157 Add a new requestID to Log fields Pull request opened by EngHabu TL;DR Add a request ID contextutils fiels.

ctx = contextutils.WithRequestID(ctx, "request-123")
logger.Infof(ctx, "this happened")

"this happened"

will be tagged with the request id Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☑︎ Any pending items have an associated Issue Tracking Issue fixes flyteorg/flyte#3577 flyteorg/flytestdlib Codecov: 67.96% (-0.21%) compared to 0dbe3c2 ✅ 6 other checks have passed 6/7 successful checks

#1585 Unify sqlalchemy Dockerfiles Pull request opened by eapolinario TL;DR ditto Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Similar to #1246 Tracking Issue https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit GitHub Actions: lint GitHub Actions: docs GitHub Actions: build-plugins (3.11, flytekit-sqlalchemy) GitHub Actions: build-plugins (3.11, flytekit-snowflake) GitHub Actions: build-plugins (3.11, flytekit-polars) GitHub Actions: build-plugins (3.11, flytekit-papermill) GitHub Actions: build-plugins (3.11, flytekit-pandera) GitHub Actions: build-plugins (3.11, flytekit-kf-tensorflow) GitHub Actions: build-plugins (3.11, flytekit-kf-pytorch) GitHub Actions: build-plugins (3.11, flytekit-kf-mpi) GitHub Actions: build-plugins (3.11, flytekit-k8s-pod) GitHub Actions: build-plugins (3.11, flytekit-dolt) GitHub Actions: build-plugins (3.11, flytekit-deck-standard) GitHub Actions: build-plugins (3.11, flytekit-dbt) GitHub Actions: build-plugins (3.10, flytekit-vaex) ✅ 15 other checks have passed 15/30 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flytekit/commit/e3cee8326c99ed34751752c922a5852ae38e43ac|e3cee832>

- Unify sqlalchemy Dockerfiles (#1585) flyteorg/flytekit

#1586 Bump tensorflow from 2.10.0 to 2.11.1 in /plugins/flytekit-mlflow Pull request opened by dependabot[bot] Bumps tensorflow from 2.10.0 to 2.11.1. Release notes Sourced from tensorflow's releases.

TensorFlow 2.11.1

Release 2.11.1

Note: TensorFlow 2.10 was the last TensorFlow release that supported GPU on native-Windows. Starting with TensorFlow 2.11, you will need to install TensorFlow in WSL2, or install tensorflow-cpu and, optionally, try the TensorFlow-DirectML-Plugin.

• Security vulnerability fixes will no longer be patched to this Tensorflow version. The latest Tensorflow version includes the security vulnerability fixes. You can update to the latest version (recommended) or patch security vulnerabilities yourself steps. You can refer to the release notes of the latest Tensorflow version for a list of newly fixed vulnerabilities. If you have any questions, please create a GitHub issue to let us know.

This release also introduces several vulnerability fixes:

• Fixes an FPE in TFLite in conv kernel CVE-2023-27579

• Fixes a double free in Fractional(Max/Avg)Pool CVE-2023-25801

• Fixes a null dereference on ParallelConcat with XLA CVE-2023-25676

• Fixes a segfault in Bincount with XLA CVE-2023-25675

• Fixes an NPE in RandomShuffle with XLA enable CVE-2023-25674

• Fixes an FPE in TensorListSplit with XLA CVE-2023-25673

• Fixes segmentation fault in tfg-translate CVE-2023-25671

• Fixes an NPE in QuantizedMatMulWithBiasAndDequantize CVE-2023-25670

• Fixes an FPE in AvgPoolGrad with XLA CVE-2023-25669

• Fixes a heap out-of-buffer read vulnerability in the QuantizeAndDequantize operation CVE-2023-25668

• Fixes a segfault when opening multiframe gif CVE-2023-25667

• Fixes an NPE in SparseSparseMaximum CVE-2023-25665

• Fixes an FPE in AudioSpectrogram CVE-2023-25666

• Fixes a heap-buffer-overflow in AvgPoolGrad CVE-2023-25664

• Fixes a NPE in TensorArrayConcatV2 CVE-2023-25663

• Fixes a Integer overflow in EditDistance CVE-2023-25662

• Fixes a Seg fault in

tf.raw_ops.Print

CVE-2023-25660

• Fixes a OOB read in DynamicStitch CVE-2023-25659

• Fixes a OOB Read in GRUBlockCellGrad CVE-2023-25658

TensorFlow 2.11.0

Release 2.11.0

Breaking Changes

• The

tf.keras.optimizers.Optimizer

base class now points to the new Keras optimizer, while the old optimizers have been moved to the

tf.keras.optimizers.legacy

namespace.

If you find your workflow failing due to this change, you may be facing one of the following issues:

• Checkpoint loading failure. The new optimizer handles optimizer state differently from the old optimizer, which simplifies the logic of checkpoint saving/loading, but at the cost of breaking checkpoint backward compatibility in some cases. If you want to keep using an old checkpoint, please change your optimizer to

<http://tf.keras.optimizer.legacy.XXX|tf.keras.optimizer.legacy.XXX>

(e.g.

tf.keras.optimizer.legacy.Adam

).

• TF1 compatibility. The new optimizer,

tf.keras.optimizers.Optimizer

, does not support TF1 any more, so please use the legacy optimizer

<http://tf.keras.optimizer.legacy.XXX|tf.keras.optimizer.legacy.XXX>

. We highly recommend migrating your workflow to TF2 for stable support and new features.

• Old optimizer API not found. The new optimizer,

tf.keras.optimizers.Optimizer

, has a different set of public APIs from the old optimizer. These API changes are mostly related to getting rid of slot variables and TF1 support. Please check the API documentation to find alternatives to the missing API. If you must call the deprecated API, please change your optimizer to the legacy optimizer.

• Learning rate schedule access. When using a

tf.keras.optimizers.schedules.LearningRateSchedule

, the new optimizer's

learning_rate

property returns the current learning rate value instead of a

LearningRateSchedule

object as before. If you need to access the

LearningRateSchedule

object, please use

optimizer._learning_rate

.

• If you implemented a custom optimizer based on the old optimizer. Please set your optimizer to subclass

<http://tf.keras.optimizer.legacy.XXX|tf.keras.optimizer.legacy.XXX>

. If you want to migrate to the new optimizer and find it does not support your optimizer, please file an issue in the Keras GitHub repo.

• Errors, such as

Cannot recognize variable...

. The new optimizer requires all optimizer variables to be created at the first

apply_gradients()

or

minimize()

call. If your workflow calls the optimizer to update different parts of the model in multiple stages, please call

optimizer.build(model.trainable_variables)

before the training loop.

• Timeout or performance loss. We don't anticipate this to happen, but if you see such issues, please use the legacy optimizer, and file an issue in the Keras GitHub repo.

The old Keras optimizer will never be deleted, but will not see any new feature additions. New optimizers (for example,

tf.keras.optimizers.Adafactor

) will only be implemented based on the new

tf.keras.optimizers.Optimizer

base class.

•

tensorflow/python/keras

code is a legacy copy of Keras since the TensorFlow v2.7 release, and will be deleted in the v2.12 release. Please remove any import of

tensorflow.python.keras

and use the public API with

from tensorflow import keras

or

import tensorflow as tf; tf.keras

.

Major Features and Improvements

... (truncated) Changelog Sourced from tensorflow's changelog.

Release 2.11.1

Note: TensorFlow 2.10 was the last TensorFlow release that supported GPU on native-Windows. Starting with TensorFlow 2.11, you will need to install TensorFlow in WSL2, or install tensorflow-cpu and, optionally, try the TensorFlow-DirectML-Plugin.

• Security vulnerability fixes will no longer be patched to this Tensorflow version. The latest Tensorflow version includes the security vulnerability fixes. You can update to the latest version (recommended) or patch security vulnerabilities yourself steps. You can refer to the release notes of the latest Tensorflow version for a list of newly fixed vulnerabilities. If you have any questions, please create …

flyteorg/flytekit GitHub Actions: docs GitHub Actions: Docs Warnings ✅ 2 other checks have passed 2/4 successful checks

#1401 [wip] Backend plugin system Pull request opened by kumare3 on <!date^1672789750^{date_short}|2023-01-03T23:49:10Z> TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit Codecov: 0.00% of diff hit (target 68.57%) GitHub Actions: build-plugins (3.10, flytekit-bigquery) ✅ 28 other checks have passed 28/30 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytesnacks/tree/master|master>

by samhita-alla

<https://github.com/flyteorg/flytesnacks/commit/3bb8e2e1ce26a754568a2a8ab9d80fb972456b7b|3bb8e2e1>

- Add reference launch plan example (#977) flyteorg/flytesnacks

Release - flytesnacks v0.3.190 New release published by flyte-bot Changelog • 3bb8e2e Add reference launch plan example (#977) • fa89202 clean up warnings in getting started docs (#981) flyteorg/flytesnacks

1 new commit pushed to

<https://github.com/flyteorg/flyteplugins/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flyteplugins/commit/a3f73343d883c8ae39b22757f90f4f3bb290d50a|a3f73343>

- Inject container resource during BuildRawContainer (#335) flyteorg/flyteplugins

Release - v1.0.47 New release published by flyte-bot Changelog • a3f7334 Inject container resource during BuildRawContainer (#335) flyteorg/flyteplugins

1 new commit pushed to

<https://github.com/flyteorg/flytepropeller/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flytepropeller/commit/986f014ae69505353833c20a38d4a58094b800cc|986f014a>

- moved controller runtime start out of webhook Run function (#546) flyteorg/flytepropeller

Release - v1.1.77 New release published by flyte-bot Changelog • 986f014 moved controller runtime start out of webhook Run function (#546) flyteorg/flytepropeller