#537 Validate user-provided entity versions are url safe Pull request opened by katrogan TL;DR Ensures that user-provided entity version are url safe since they are used http paths to manipulate the entities and can lead to 404s when they include slashes. Execution names, which are also user-provided and used in http paths are already constrained to a smaller subset of allowable chars Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☐ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Encountered with a user Tracking Issue fixes flyteorg/flyte#3427 Follow-up issue NA flyteorg/flyteadmin GitHub Actions: Build & Push Flyteadmin Image GitHub Actions: Goreleaser GitHub Actions: Build & Push Flytescheduler Image GitHub Actions: Bump Version ✅ 8 other checks have passed 8/12 successful checks

#1526 Data subsystem Pull request opened by wild-endeavor TL;DR This replaces the flytekit custom persistence layer and routes all calls to fsspec directly. Currently still relying on

s3fs/gcsfs

to provide the fsspec implementations. Important This PR removes the

DataPersistence

and

DataPersistencePlugins

constructs completely. If you were importing these, these are now going away. However most people should not have been using these. The

flytekitplugins-data-fsspec

plugin has also been emptied out, though it will continue to be published for the time being. The logic that was in there originally has been moved into flytekit code in this PR, including the Structured Dataset handlers. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Notes for users and contributors If you're importing

LOCAL

or

S3

from

flytekit/types/structured/structured_dataset.py

these have been removed. If you're registering with

None

as the protocol for a structured dataset handler, what happens now is that the handler is registered with the

fsspec

protocol. For users that are not familiar with

StructuredDataset

encoders and decoders, which one gets used is based on three things: the Python type of the dataclass, the storage protocol (local vs s3 vs gcs etc.) and the file format (parquet, csv, etc.). This was done as a separate commit #1543. The waterfall is now: 1. Exact match on all three attributes 2. Protocol match but a generic format (

""

) handler, if registered 3. Protocol match and a match on the default format for that dataframe type, if default format is set 4. An

fsspec

handler with a format match (will match generic

""

format too) 5. An

fsspec

handler with a generic format, if registered 6. An

fsspec

handler with the default format for that dataframe type, if default format is set 7. An

fsspec

handler if that protocol only has one entry and the requested format was generic. 8. Protocol specific handler if that protocol only has one entry. Others • Added a

Dockerfile.dev

for slightly easier debugging. (Longer term we hope to come up with a better dev experience.) • Some attempt was made to add the ability to include a custom header containing a hash to the fsspec HTTP filesystem. This is needed for

pyflyte run

register

and anything else that uses the

DataProxy

service to post upload to a signed URL. Unfortunately no solution was found, so the upload path for these two commands just uses the old way of calling the requests library. Tracking Issue flyteorg/flyte#3197 flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteidl/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flyteidl/commit/3dfcaf6671d85ee72c1ce00961c17421c5c91111|3dfcaf66>

- Init customTokenSource.refreshTime (#381) flyteorg/flyteidl

Release - v1.3.11 New release published by flyte-bot Changelog • 3dfcaf6 Init customTokenSource.refreshTime (#381) flyteorg/flyteidl

#207 [Snyk] Upgrade com.google.api.grpc:proto-google-common-protos from 2.14.0 to 2.14.1 Pull request opened by snyk-bot Snyk has created this PR to upgrade com.google.api.grpc:proto-google-common-protos from 2.14.0 to 2.14.1. ℹ️ Keep your dependencies up-to-date. This makes it easier to fix existing vulnerabilities and to more quickly identify and fix newly disclosed vulnerabilities when they affect your project. * * * • The recommended version is 1 version ahead of your current version. • The recommended version was released 22 days ago, on 2023-02-15. * * * Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open upgrade PRs. For more information: https://camo.githubusercontent.com/5e2f7e5c1cc930c95558c8bca957e96cce37c4a530c9a0ac4bc885372390e668/68747470733a2f2f6170692e7365676d656e742e696f2f76312f706978656c2f747261636b3f646174613d65794a33636d6c305a55746c65534936496e4a79576d785a634564485932527954485a7362306c596430645563566734576b4652546e4e434f5545774969776959573576626e6c746233567a535751694f694935596a497a5a4751354e4330354e444d7a4c5451774d6a6374595441354e5330794e5745344f446b324d6a4a6d5a4459694c434a6c646d567564434936496c425349485a705a58646c5a434973496e42796233426c636e52705a584d694f6e736963484a4a5a434936496a6c694d6a4e6b5a446b304c546b304d7a4d744e4441794e7931684d446b314c544931595467344f5459794d6d5a6b4e694a3966513d3d 🧐 View latest project report 🛠️ Adjust upgrade PR settings 🔕 Ignore this dependency or unsubscribe from future upgrade PRs flyteorg/flytekit-java ✅ All checks have passed 3/3 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by kumare3

<https://github.com/flyteorg/flytekit/commit/e90ee2576c87298826b1ba702cf7569f43f737ba|e90ee257>

- Pyflyte docs formatting (#1538) flyteorg/flytekit

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by wild-endeavor

<https://github.com/flyteorg/flytekit/commit/28da983bba36e243bc671f9ba1aa53a0791efd62|28da983b>

- Data subsystem (#1526) flyteorg/flytekit

#208 Add scala subworkflow ex Pull request opened by sonjaer Read then delete this section - Make sure to use a concise title for the pull-request. _- Use #patch, #minor or #major in the pull-request title to bump the corresponding version. Otherwise, the patch version will be bumped. More details_ TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit-java GitHub Actions: build-snapshot DCO: DCO ✅ 1 other check has passed 1/3 successful checks

#209 Add ability to fetch lp with version Pull request opened by sonjaer TL;DR Factory method to create remote launch plan with fixed version Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added - NA there are already tests for it ☐ Code documentation added NA ☐ Any pending items have an associated Issue NA flyteorg/flytekit-java ✅ All checks have passed 3/3 successful checks

#538 Bumping fosite version to use community maintained JWT implementation Pull request opened by hamersaw TL;DR The existing JWT implementation is no longer maintained and has a existing security CVE of HIGH. By increasing the fosite version we instead use the community maintained, backwards compatible JWT implementation which has no known security issues. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue fixes flyteorg/flyte#3368 Follow-up issue NA flyteorg/flyteadmin ✅ All checks have passed 2/2 successful checks

#101 Replaced github.com/satori/go.uuid with github.com/gofrs/uuid Pull request opened by hamersaw TL;DR Removing the dependency on github.com/satori/go.uuid which is no longer a maintained library and has known security issues. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue fixes flyteorg/flyte#3366 Follow-up issue NA flyteorg/datacatalog GitHub Actions: Goreleaser GitHub Actions: Build & Push Datacatalog Image GitHub Actions: Bump Version ✅ 6 other checks have passed 6/9 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by sonjaer

<https://github.com/flyteorg/flytekit-java/commit/2aaab0b28221fc43b87bbf96593af90008e3c1ce|2aaab0b2>

- add scala subworkflow ex (#208) flyteorg/flytekit-java

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by sonjaer

<https://github.com/flyteorg/flytekit-java/commit/116814abe545532633dda63117c00d0426f422f0|116814ab>

- Add ability to fetch lp with version (#209) flyteorg/flytekit-java

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by github-actions[bot]

<https://github.com/flyteorg/flytekit-java/commit/6ae23c45743a60012868438a77ebef4f622a573e|6ae23c45>

- [maven-release-plugin] prepare release 0.4.6 flyteorg/flytekit-java

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by github-actions[bot]

<https://github.com/flyteorg/flytekit-java/commit/aa72cfb18cf9f78d5f4eb093da146b8651b92578|aa72cfb1>

- [maven-release-plugin] prepare for next development iteration flyteorg/flytekit-java

Release - 0.4.6 New release published by sonjaer • Add ability to fetch lp with version #209 • Add scala subworkflow ex #208 flyteorg/flytekit-java

#3441 [BUG] Type Engine Resolution should properly support inheritance Issue created by nicholasjng Describe the bug Hello Flyte team, I am trying to define a generic Python type wrapping a single resource (think data frame, string etc.) that is supposed to inherit serialization features from the builtin

FlyteFile

. Everything went well until I tried to define a custom

TypeTransformer

for it, which I wanted to behave exactly like `FlyteFile`'s type transformer with a few additions. Pseudocode:

class Resource(Generic[T], FlyteFile):
    ...

class ResourceTransformer(TypeTransformer[Resource]):
    def to_literal(ctx, python_val, python_type, expected):
        lit = TypeEngine().to_literal(ctx, python_val, FlyteFile, None)  # <- construct a literal representation of the FlyteFile
        ... # now, do something with the literal like embedding a hash etc.
        return lit

TypeEngine.register(ResourceTransformer(...))

The problem is that the current type resolution will now resolve

FlyteFile

as the appropriate transformer for a

Resource

object, as per this code: https://github.com/flyteorg/flytekit/blob/28da983bba36e243bc671f9ba1aa53a0791efd62/flytekit/core/type_engine.py#L756-L760 There is a TODO comment a little above that sort of acknowledges this problem, and proposes to use an ordered dict, presumably to walk it in reverse and resolve the

Resource

transformer before the

FlyteFile

transformer. This might be brittle though for multiple inheritance use cases like the one here, maybe even with a third resource type sprinkled in that gets registered way after the other two. What I propose instead is trying to walk down the `python_val`'s class hierarchy and greedily match the first type transformer available, roughly like so:

# flytekit/core/type_engine.py
# begins at the code block linked above

# Step 
# To facilitate cases where users may specify one transformer for multiple types that all inherit from 
# parent.
if hasattr(python_type, "__mro__"):
   for base_type in inspect.getmro(python_type):
         try:
             return cls._REGISTRY[base_type]
         except KeyError:
             continue

For an inheritance chain like mine with MRO

Resource -> FlyteFile -> ... -> object

, this would lead to a match directly at the

Resource

level. What do you think? Expected behavior

TypeEngine().to_literal(ctx, python_val, python_type, expected)

calls the

ResourceTransformer

class to construct the literal, but instead it calls `FlyteFile`'s associated transformer. Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#719 feat: differentiate between cache disabled and cache put failure Pull request opened by james-union flyteorg/flyte#3430 (comment) I've added this icon (pointing with red) Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes flyteorg/flyte#3430 flyteorg/flyteconsole Codecov: 66.85% (-0.11%) compared to ffb161c GitHub Actions: Build & Push FlyteConsole Image GitHub Actions: Get Release Tag GitHub Actions: Generate Release ✅ 7 other checks have passed 7/11 successful checks

@Yee has left the channel

#539 Add Dan Rammer and Eduardo to codeowners Pull request opened by eapolinario TL;DR Add members of the OSS team to CODEOWNERS file flyteorg/flyteadmin GitHub Actions: Goreleaser GitHub Actions: Build & Push Flytescheduler Image GitHub Actions: Build & Push Flyteadmin Image GitHub Actions: Bump Version ✅ 10 other checks have passed 10/14 successful checks

#1548 [v1.3 Backport] Filter out remote entity when generating pb (#1545) Pull request opened by wild-endeavor Backport #1545 flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteadmin/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flyteadmin/commit/2d3942298bbbdcd9841be002612ba0c4da77e5eb|2d394229>

- Add Dan Rammer and Eduardo to codeowners (#539) flyteorg/flyteadmin

#3444 Fix local compile Pull request opened by jeevb flyteorg/flyte ✅ All checks have passed 6/6 successful checks

#1549 [v1.2 Backport] be24c52 through e90ee25 Pull request opened by wild-endeavor Backport be24c52 through e90ee25 flyteorg/flytekit GitHub Actions: build-plugins (3.8, flytekit-ray) GitHub Actions: build-plugins (3.8, flytekit-polars) GitHub Actions: build-plugins (3.8, flytekit-papermill) GitHub Actions: build-plugins (3.8, flytekit-pandera) GitHub Actions: build-plugins (3.8, flytekit-onnx-tensorflow) GitHub Actions: build-plugins (3.8, flytekit-onnx-scikitlearn) GitHub Actions: build-plugins (3.8, flytekit-onnx-pytorch) GitHub Actions: build-plugins (3.8, flytekit-modin) GitHub Actions: build-plugins (3.8, flytekit-kf-tensorflow) GitHub Actions: build-plugins (3.8, flytekit-kf-pytorch) GitHub Actions: build-plugins (3.8, flytekit-kf-mpi) GitHub Actions: build-plugins (3.8, flytekit-k8s-pod) GitHub Actions: build-plugins (3.8, flytekit-hive) GitHub Actions: build-plugins (3.8, flytekit-greatexpectations) GitHub Actions: build-plugins (3.8, flytekit-duckdb) GitHub Actions: build-plugins (3.8, flytekit-dolt) GitHub Actions: build-plugins (3.8, flytekit-deck-standard) GitHub Actions: build-plugins (3.8, flytekit-dbt) GitHub Actions: build-plugins (3.8, flytekit-data-fsspec) GitHub Actions: build-plugins (3.8, flytekit-bigquery) GitHub Actions: build-plugins (3.8, flytekit-aws-sagemaker) GitHub Actions: build-plugins (3.8, flytekit-aws-batch) GitHub Actions: build-plugins (3.8, flytekit-aws-athena) GitHub Actions: build (windows-latest, 3.9) GitHub Actions: build (windows-latest, 3.8) GitHub Actions: build (windows-latest, 3.7) GitHub Actions: build (ubuntu-latest, 3.10) GitHub Actions: build (ubuntu-latest, 3.9) GitHub Actions: build (ubuntu-latest, 3.8) ✅ 1 other check has passed 1/30 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by jeevb

<https://github.com/flyteorg/flyte/commit/401973c72dd59497f03ca6efd2267fe850ccb5c3|401973c7>

- Fix local compile (#3444) flyteorg/flyte

#3197 [Core feature] [flytekit] use pyarrow + fsspec plugin by default Issue created by kumare3 Motivation: Why do you think this is important? Currently we have our own datapersistence layer. This is because we wanted to support aws cli and gsutil. I think we should drop support for aws cli and gsutil and completely migrate to using datafusion / arrow file system or fsspec On reading I see that the datafusion project has matured and has a fantastic rust object_Store crate which works great for Flytekit https://arrow.apache.org/docs/python/filesystems.html Goal: What should the final outcome look like, ideally? Flyte is enriched with the amazing integrations of arrow and the great growing ecosystem. Describe alternatives you've considered NA Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

Release - [v1.3] Do not register remote entities New release published by wild-endeavor What's Changed • [v1.3 Backport] Filter out remote entity when generating pb (#1545) by @wild-endeavor in #1548 Full Changelog: v1.3.3...v1.3.4 flyteorg/flytekit

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by pingsutw

<https://github.com/flyteorg/flyte/commit/05771f335998ec19a4cf032b4419fd894bd1f4e0|05771f33>

- Update swagger.rst (#3398) flyteorg/flyte

#3445 Update Flyte components Pull request opened by flyte-bot Updated flyte deployment • Updated GCP Flyte kustomize generated manifest file • Updated EKS Flyte kustomize generated manifest file • Updated Sandbox Flyte kustomize generated manifest file • Updated TEST Flyte kustomize generated manifest file • Updated GCP Flyte helm generated manifest file • Updated EKS Flyte helm generated manifest file • Updated Sandbox Flyte helm generated manifest file • Updated TEST Flyte helm generated manifest file • Auto-generated by [flyte-bot] flyteorg/flyte GitHub Actions: generate_kustomize GitHub Actions: docs GitHub Actions: trigger-sandbox-build GitHub Actions: generate_helm GitHub Actions: Functional test GitHub Actions: compile GitHub Actions: test-bootstrap GitHub Actions: trigger-sandbox-lite-build ✅ 2 other checks have passed 2/10 successful checks

#368 added DataLoadingConfig to K8sPod message Pull request opened by hamersaw TL;DR This PR adds the

DataLoadingConfig

information to the

K8sPod

proto message so that Flyte can inject copilot as a sidecar in Pod tasks. This allows the use of

PodTemplate

configuration with

ContainerTasks

. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue flyteorg/flyte#3123 Follow-up issue NA flyteorg/flyteidl ✅ All checks have passed 13/13 successful checks