Release - v1.1.70 New release published by flyte-bot Changelog • 1ccd59c Forward all claims in userinfo response (#511) flyteorg/flyteadmin

#3228 Add Kubernetes objects for dev mode with single-binary sandbox Pull request opened by jeevb on <!date^1673410264^{date_short}|2023-01-11T04:11:04Z> This PR completes support for running Flyte locally against a sandbox cluster running on your machine. Specifically, it adds support for Propeller's secret webhook. It does so by creating a headless service pointing to the docker host's IP address. The host IP is injected via the

%{HOST_GATEWAY_IP}%

template variable. flyteorg/flyte GitHub Actions: trigger-sandbox-lite-build GitHub Actions: trigger-single-binary-build GitHub Actions: compile GitHub Actions: Functional test ✅ 6 other checks have passed 6/10 successful checks

#164 Change SdkBindingData to be typed and add typed transform outputs #minor Pull request opened by sonjaer TL;DR This PR tries to accomplish two features: Change the SdkBindingData to be typed SdkBindingData<?> This change allows the user to know the inner type of the attributes from INPUT and OUTPUT at development time. Add typed transform outputs This change allows getting the outputs of a SdkNode.getOutputs() this output is a typed output using an AutoValue class or a case class. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☐ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Change the SdkBindingData to be typed SdkBindingData<?> This change allows the user to know the inner type of the attributes from INPUT and OUTPUT at development time. Before this change, the users must go to the input/output class to figure out the specified attribute type. Now, the SdkBindingData<?> shows the inner type value making the user experience more smoothie. Add typed transform outputs This change allows getting the outputs of a SdkNode.getOutputs() this output is a typed output using an AutoValue class or a case class. These changes force all the input/output class attributes to be SdkBindingData<?> and now you need to do a

SdkBindingData.get()

to get the inner value in the run task context (java example, scala example), but at the same time these changes allow to recover the attributes by name in the workflow site (java example, scala example). Tracking Issue • flyteorg/flyte#3250 • flyteorg/flyte#3251 Follow-up issue • flyteorg/flyte#3252 flyteorg/flytekit-java ✅ All checks have passed 3/3 successful checks

#1408 Return error code on fail Pull request opened by pingsutw Signed-off-by: Kevin Su pingsutw@gmail.com TL;DR aws batch get the status of job from return code. Therefore, we should return error code once we catch the error. blocked by flyteorg/flyteplugins#306 Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Tracking Issue https://flyte-org.slack.com/archives/C01P3B761A6/p1664972219043289 flyteorg/flyte#2979 flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

#517 use different perm Pull request opened by wild-endeavor Signed-off-by: Yee Hing Tong wild-endeavor@users.noreply.github.com Read then delete this section - Make sure to use a concise title for the pull-request. _- Use #patch, #minor or #major in the pull-request title to bump the corresponding version. Otherwise, the patch version will be bumped. More details_ TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytepropeller

1 new commit pushed to

<https://github.com/flyteorg/flytepropeller/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flytepropeller/commit/ce57dbf15274a77c7037f367bda738e6ce33b453|ce57dbf1>

- use different perm (#517) flyteorg/flytepropeller

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flyte/commit/782fe745452a07294db47f6ff782e5c559ac3c92|782fe745>

- Add dask operator (#3145) flyteorg/flyte

#3209 Changelog for 1.3 Pull request opened by wild-endeavor on <!date^1672871808^{date_short}|2023-01-04T22:36:48Z> Just the changelog, no code changes. flyteorg/flyte GitHub Actions: generate_kustomize GitHub Actions: compile GitHub Actions: docs ✅ 3 other checks have passed 3/6 successful checks

Release - v1.1.62 New release published by flyte-bot Changelog • ce57dbf use different perm (#517) flyteorg/flytepropeller

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by jeevb

<https://github.com/flyteorg/flyte/commit/c674bfa31ef0d262112b083950c4425480a18bc3|c674bfa3>

- Add Kubernetes objects for dev mode with single-binary sandbox (#3228) flyteorg/flyte

#380 change host Pull request opened by wild-endeavor Signed-off-by: Yee Hing Tong wild-endeavor@users.noreply.github.com Read then delete • Make sure to use a concise title for the pull-request. • Use #patch, #minor #majora or #none in the pull-request title to bump the corresponding version. Otherwise, the patch version will be bumped. More details TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytectl

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flyte/commit/c19c282df236b4f7917a64f7928eeaa121723b56|c19c282d>

- Changelog for 1.3 (#3209) flyteorg/flyte

#3230 Update Flyte components Pull request opened by flyte-bot Updated flyte deployment • Updated GCP Flyte kustomize generated manifest file • Updated EKS Flyte kustomize generated manifest file • Updated Sandbox Flyte kustomize generated manifest file • Updated TEST Flyte kustomize generated manifest file • Updated GCP Flyte helm generated manifest file • Updated EKS Flyte helm generated manifest file • Updated Sandbox Flyte helm generated manifest file • Updated TEST Flyte helm generated manifest file • Auto-generated by [flyte-bot] flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytectl/tree/master|master>

by wild-endeavor

<https://github.com/flyteorg/flytectl/commit/b0d98931bf8780c193f9fc101242309058138795|b0d98931>

- Change extra host to host-gateway (#380) flyteorg/flytectl

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flyte/commit/f69fb09ca189e8bf57e1a6a12db168274f640d15|f69fb09c>

- Update Flyte components (#3230) flyteorg/flyte

Release - v0.6.26 New release published by flyte-bot Changelog • b0d9893 Change extra host to host-gateway (#380) flyteorg/flytectl

1 new commit pushed to

<https://github.com/flyteorg/homebrew-tap/tree/main|main>

by flyte-bot

<https://github.com/flyteorg/homebrew-tap/commit/6d942c75f7f2a62d6180fb5d16c454d6e28c8d54|6d942c75>

- Brew formula update for flytectl version v0.6.26 flyteorg/homebrew-tap

#151 auto-update release Pull request opened by flyte-bot Automated changes by create-pull-request GitHub action flyteorg/flyteorg.github.io

Release - Flyte v1.3.0 milestone release New release published by flyte-bot Flyte v1.3.0 The main features of this 1.3 release are • Databricks support as part of the Spark plugin • New Helm chart that offers a simpler deployment using just one Flyte service • Signaling/gate node support (human in the loop tasks) • User documentation support (backend and flytekit only, limited types) The latter two are pending some work in Flyte console, they will be piped through fully by the end of Q1. Support for setting and approving gate nodes is supported in

FlyteRemote

however, though only a limited set of types can be passed in. Notes There are a couple things to point out with this release. Caching on Structured Dataset Please take a look at the flytekit PR notes for more information but if you haven't bumped Propeller to version v1.1.36 (aka Flyte v1.2) or later, tasks that take as input a dataframe or a structured dataset type, that are cached, will trigger a cache miss. If you've upgraded Propeller, it will not. Flytekit Remote Types In the

FlyteRemote

experience, fetched tasks and workflows will now be based on their respective "spec" classes in the IDL (task/wf) rather than the template. The spec messages are a superset of the template messages so no information is lost. If you have code that was accessing elements of the templates directly however, these will need to be updated. Usage Overview Databricks Please refer to the documentation for setting up Databricks. Databricks is a subclass of the Spark task configuration so you'll be able to use the new class in place of the more general

Spark

configuration.

from flytekitplugins.spark import Databricks
@task(
    task_config=Databricks(
        spark_conf={
            "spark.driver.memory": "1000M",
            "spark.executor.memory": "1000M",
            "spark.executor.cores": "1",
            "spark.executor.instances": "2",
            "spark.driver.cores": "1",
        },
        databricks_conf={
            "run_name": "flytekit databricks plugin example",
            "new_cluster": {
                "spark_version": "11.0.x-scala2.12",
                "node_type_id": "r3.xlarge",
                "aws_attributes": {
                    "availability": "ON_DEMAND",
                    "instance_profile_arn": "arn:aws:iam::1237657460:instance-profile/databricks-s3-role",
                },
                "num_workers": 4,
            },
            "timeout_seconds": 3600,
            "max_retries": 1,
        }
    ))

New Deployment Type A couple releases ago, we introduced a new Flyte executable that combined all the functionality of Flyte's backend into one command. This simplifies the deployment in that only one image needs to run now. This approach is now our recommended way for new comers to the project to install and administer Flyte and there is a new Helm chart also. Documentation has been updated to take this into account. For new installations of Flyte, clusters that do not already have the

flyte-core

or

flyte

charts installed, users can

helm install flyte-server flyteorg/flyte-binary --namespace flyte --values your_values.yaml

New local demo environment Users may have noticed that the environment provided by

flytectl demo start

has also been updated to use this new style of deployment, and internally now installs this new Helm chart. The demo cluster now also exposes an internal docker registry on port

30000

. That is, with the new demo cluster up, you can tag and push to

localhost:30000/yourimage:tag123

and the image will be accessible to the internal Docker daemon. The web interface is still at

localhost:30080

, Postgres has been moved to

30001

and the Minio API (not web server) has been moved to

30002

. Human-in-the-loop Workflows Users can now insert sleeps, approval, and input requests, in the form of gate nodes. Check out one of our earlier issues for background information.

from flytekit import wait_for_input, approve, sleep

@workflow
def mainwf(a: int):
    x = t1(a=a)
    s1 = wait_for_input("signal-name", timeout=timedelta(hours=1), expected_type=bool)
    s2 = wait_for_input("signal name 2", timeout=timedelta(hours=2), expected_type=int)
    z = t1(a=5)
    zzz = sleep(timedelta(seconds=10))
    y = t2(a=s2)
    q = t2(a=approve(y, "approvalfory", timeout=timedelta(hours=2)))
    x >> s1
    s1 >> z
    z >> zzz
    ...

These also work inside

@dynamic

tasks. Interacting with signals from flytekit's remote experience looks like

from flytekit.remote.remote import FlyteRemote
from flytekit.configuration import Config
r = FlyteRemote(
    Config.auto(config_file="/Users/ytong/.flyte/dev.yaml"),
   default_project="flytesnacks",
   default_domain="development",
)
r.list_signals("atc526g94gmlg4w65dth")
r.set_signal("signal-name", "execidabc123", True)

Overwritten Cached Values on Execution Users can now configure workflow execution to overwrite the cache. Each task in the workflow execution, regardless of previous cache status, will execute and write cached values - overwritting previous values if necessary. This allows previously corrupted cache values to be corrected without the tedious process of incrementing the

cache_version

and re-registering Flyte workflows / tasks. Support for Dask Users will be able to spawn Dask ephemeral clusters as part of their workflows, similar to the support for Ray and Spark. Looking Ahead In the coming release, we are focusing on... 1. Out of core plugin: Make backend plugin scalable and easy to author. No need of code generation, using tools that MLEs and Data Scientists are not accustomed to using. 2. Performance Observability: We have made great progress on exposing both finer-grained runtime metrics and Flytes orchestration metrics. This is important to better understand workflow evaluation performance and mitigate inefficiencies thereof. flyteorg/flyte

Deployment to github-pages by flyte-bot flyteorg/flyte

#3232 [Core feature] Pass inputs inline with execution event data Issue created by katrogan Motivation: Why do you think this is important? Follow-up to #1327. For a distributed propeller deployment it's useful to pass executions inputs inline rather than as an offloaded URI. Goal: What should the final outcome look like, ideally? Raw inputs should be (configurably) sent inline in execution events Describe alternatives you've considered Current implementation Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3233 [BUG] Flyte binary not capturing go context in cluster manager Issue created by wild-endeavor Describe the bug Flyte has a series of backend services (propeller, admin, etc.). The Flyte binary, built here, combines all these into one. When we wrote it, we didn't realize the behavior of the

Wait()

call on the

errgroup.WithContext

doesn't return on first error. It waits til all the go-routines finish, and then returns the first error. I only cancels the context. This means that all the goroutines have to monitor the context and terminate itself on context cancel. This is missing for the clusterresourcemanager - flyte/cmd/single/start.go Line 49 in </flyteorg/flyte/commit/f69fb09ca189e8bf57e1a6a12db168274f640d15|f69fb09> . This means that if Flyte Admin fails, the binary itself will proceed as normal, just without admin, since the cluster resource manager runs in perpetuity, which is how we found this issue. Expected behavior The whole process should fail if Admin fails. Add context monitoring logic to cluster resource controller and go through the other errgroups to make sure everything launched is also doing the right thing. Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flytekit/commit/531db3cbadda1e19a42b307a989106db35a836c0|531db3cb>

- Add dask plugin #patch (#1366) flyteorg/flytekit

#1366 Add dask plugin #patch Pull request opened by bstadlbauer TL;DR This adds the Python side of the

dask

backend plugin (

flytekitplugins-dask

) Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☑︎ Any pending items have an associated Issue Complete description This adds the Python side of the

dask

backed plugin (links to the related PRs can be found in the linked issue below). TODO before this can be merged: After flyteorg/flyteidl#339 is merged: ☑︎ Make sure

flytekit

version is up to date in

plugins/flytekit-dask/setup.py

(needs to have

buf

protobufs) ☑︎ Make sure

flyteidl

version is up to date in

plugins/flytekit-dask/setup.py

(needs to support

dask

IDL) ☑︎ Add

plugins/flytekit-dask/requirements.in

and lock (on Linux) ☑︎ Add

dask[distributed]

to

<http://doc-requirements.in|doc-requirements.in>

and re-lock (on Linux) ☐ Check to make sure documentation looks ok Tracking Issue flyteorg/flyte#427 Follow-up issue NA flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

#1409 fix: upgrade request dependency to version that does not use vulnerable py dep Pull request opened by mbuchove fix: upgrade request dependency to version that does not use vulnerable py dep TL;DR This request is just to upgrade the

retry

package dependency in

requirements.txt

for flytekit. The original unmaintained package has a dependency on the

py

package, which has a CVE with no fix version available: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-42969 Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description In this PR, it is replaced with the recent release of the

retry2

package version

0.9.5

, which is a fork of the original

retry

package: https://pypi.org/project/retry2/ Tracking Issue flyteorg/flyte#3052 flyteorg/flytekit

#1410 Add support for overriding task configurations Pull request opened by pingsutw TL;DR Add support for overriding task configurations, but don't support changing the config type Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue https://flyte-org.slack.com/archives/CP2HDHKE1/p1670887948311139 Follow-up issue NA flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

#165 Refactor Serializer Pull request opened by rafaelraposospot TL;DR Refactor serializer and implement missing Kinds Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit-java ✅ All checks have passed 2/2 successful checks

Release - v1.3.0 New release published by eapolinario Flytekit 1.3.0 User Facing Changes -D/-d In

pyflyte

the

-d

switch was overloaded, used for both

domain

and the

destination-dir

that flytekit will extract user code into when using the fast-register feature. The latter has been updated to

-D

. This may require some users to change commands in CI/CD if the short form of these switches is used. Protobuf Support In May of last year Google announced a breaking change to the support for protobuf in python. Starting with this release we moved to this new version of the protobuf tooling, specifically, this affects some of the flytekit dependencies, including

grpcio

and

grpcio-status

. Some flytekit dependencies, namely

tensorflow

haven't upgraded yet to the new protobuf version yet (more details in tensorflow/tensorflow#53234), so for the time being, flytekit v1.2.x will be maintained for bug fixes (and possibly features as well). * * * Additional Please see Flyte 1.3 Release Notes for additional changes (Databricks, Human-in-the-loop Tasks, Dask, and FlyteRemote updates) and usage. * * * What's Changed • Sanitize query template input in sqlite task by @eapolinario in #1359 • TypeTransformer for reading and writing from TensorFlowRecord format by @ryankarlos in #1240 • Bump numpy from 1.21.6 to 1.22.0 in /tests/flytekit/integration/remote/mock_flyte_repo/workflows by @dependabot in #1170 • Update ray plugin dependency by @pingsutw in #1361 • Bring in buf protobufs by @eapolinario in #1363 • Set default format of structured dataset to empty by @pingsutw in #1159 • Adds CLI reference for pyflyte by @samhita-alla in #1362 • Update plugins requirements files versions by @eapolinario in #1365 • Signaling by @wild-endeavor in #1133 • Adding created and updated at to ExecutionClosure model by @wild-endeavor in #1371 • Restrict numpy due to deprecated aliases by @eapolinario in #1376 • Add Databricks config to Spark Job by @pingsutw in #1358 • Add overwrite_cache option the to calls of remote and local executions by @hfurkanvural in #1375 • Remove project/domain from being overridden with execution values in serialized context by @wild-endeavor in #1378 • Use TaskSpec instead of TaskTemplate for fetch_task and avoid network when loading module by @kumare3 in #1348 • Register Databricks config by @pingsutw in #1379 • PodSpec should not require primary_container name by @kumare3 in #1380 • fix(pyflyte register): change -d to -D for --destination-dir as -d is already used for --domain by @mcloney-ddm in #1381 • [Snyk] Security upgrade torch from 1.12.1 to 1.13.1 by @snyk-bot in #1374 • Handle Optional[FlyteFile] in Dataclass type transformer by @eapolinario in #1393 • Update flyte deck plugin's dependency by @pingsutw in #1395 • add FastSerializationSettings to docs by @cosmicBboy in #1386 • Added more pod tests and an example pod task by @kumare3 in #1382 • Convert default value of dict to json string in pyflyte run by @pingsutw in #1399 • docs: update register help, non-fast version is supported by @pbrogan12 in #1402 • Update log level for structured dataset by @pingsutw in #1394 • Add Niels to code owners by @pingsutw in #1404 • Signal use by @wild-endeavor in #1398 • User Documentation Proposal by @pingsutw in #1200 • Add support MLFlow plugin by @kumare3 in #1274 • fix remote API reference by @cosmicBboy in #1405 • Read structured dataset from a folder by @pingsutw in #1406 • Update default config to work out-of-the-box with flytectl demo by @cosmicBboy in #1384 • Add dask plugin #patch by @bstadlbauer in #1366 New Contributors • @hfurkanvural made their first contribution in #1375 • @mcloney-ddm made their first contribution in #1381 • @pbrogan12 made their first contribution in #1402 Full Changelog: v1.2.5...v1.3.0 flyteorg/flytekit

#3234 [Core feature] Enable project deletions Issue created by seunggs Motivation: Why do you think this is important? Currently, you can only archive a project, but it leaves a lot of hanging resources when you create unused/test projects. Goal: What should the final outcome look like, ideally? There should be a flyte API endpoint for deleting a project and all related resources completely. This allows for safe and clean removal of unused resources from flyte cluster Describe alternatives you've considered Manually deleting resources seems unsafe since most users (including myself) do not completely understand flyte internals and which resources or database entries need to be manually removed. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3235 [Docs] flytectl demo start does not work with ZFS Issue created by mficek Description I had troubles running `flytectl demo start`: it hung during initialization of k8s cluster, without any message. After examination of docker logs I realized there is an issue with k8s when running on ZFS file system. My setup is Ubuntu 22.04 with the whole hard drive partition being created as ZFS. After re-installing my system to run on ext4, everything works smoothly. It might be useful to stress this somewhere in the documentation: ZFS is not a friend the demo cluster (sandbox cluster is the same). Thanks! Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte