1 new commit pushed to

<https://github.com/flyteorg/flytepropeller/tree/master|master>

by pingsutw

<https://github.com/flyteorg/flytepropeller/commit/48c37ded438ea18aaca60b74e748760057b3a48a|48c37ded>

- Structured Dataset with generic format should be castable to Flyte Schema (#536) flyteorg/flytepropeller

Release - v1.1.70 New release published by flyte-bot Changelog • 48c37de Structured Dataset with generic format should be castable to Flyte Schema (#536) flyteorg/flytepropeller

1 new commit pushed to

<https://github.com/flyteorg/flytesnacks/tree/master|master>

by cosmicBboy

<https://github.com/flyteorg/flytesnacks/commit/3148c6c6aaf056020c523ca802439c092d472557|3148c6c6>

- Add mlflow example (#960) flyteorg/flytesnacks

#3286 [Docs] mlflow integration: create flytesnacks example page Issue created by cosmicBboy Description The MLFlow integration currently does not have an example page under the integrations section: https://github.com/flyteorg/flytesnacks/tree/master/cookbook/integrations/flytekit_plugins The purpose of this task is to add (a) a page describing the plugin and how to install it (see here) and (b) an example page on how to use it (see here) Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#967 Upgrading aws sdk to use WebIdentityTokenCredentialsProvider Pull request opened by pmahindrakar-oss Encountered permission issue when running spark history server on one of the deployments The issue being spark history server events requires to use WebIdentityTokenProvider to authenticate using the annotated role inorder to write events to the configured bucket The first version where this idneitty provider came in was in 1.11.603 which is higher than what we have in our dockerfile https://github.com/aws/aws-sdk-java/commits/master/aws-java-sdk-core/src/main/java/com/amazonaws/auth/WebIdentityTokenCredentialsProvider.java Testing this on internal tenant and saw successful events writes in the configured history server bucket flyteorg/flytesnacks GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/kubernetes/ray_example) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/kubernetes/pod) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/kubernetes/kftensorflow) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/kubernetes/kfpytorch) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/kubernetes/kfmpi) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/kubernetes/k8s_spark) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/kubernetes/k8s_dask) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/whylogs_examples) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/sql) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/papermilltasks) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/pandera_examples) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/onnx_examples) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/modin_examples) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/mlflow_example) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/greatexpectations) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/duckdb_examples) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/dolt) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/dbt_example) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/external_services/databricks) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/aws/sagemaker_training) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/aws/sagemaker_pytorch) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/aws/batch) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/aws/athena) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/core) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/case_studies/ml_training/spark_horovod) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/case_studies/ml_training/pima_diabetes) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/case_studies/ml_training/nlp_processing) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/case_studies/ml_training/mnist_classifier) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/case_studies/ml_training/house_price_prediction) GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/case_studies/feature_engineering/feast_integration) ✅ No checks have passed 0/30 successful checks

#715 chore: allow complex workflow names Pull request opened by ursucarina TL;DR The client drops wf versions that have slashes Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Added encoding of wf versions flyteorg/flyteconsole ✅ All checks have passed 2/2 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytesnacks/tree/master|master>

by pmahindrakar-oss

<https://github.com/flyteorg/flytesnacks/commit/343072b412e808e552a57f7d30bf404025b22464|343072b4>

- Upgrading aws sdk to use WebIdentityTokenCredentialsProvider (#967) flyteorg/flytesnacks

#536 Don't truncate error messages when listing task executions Pull request opened by katrogan TL;DR Unfortunately the current behavior truncates the very useful error message shown in the sidebar for each task execution attempt. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Discovered after updating tenant internally Tracking Issue flyteorg/flyte#3337 Follow-up issue NA flyteorg/flyteadmin GitHub Actions: End2End Test / End to End tests GitHub Actions: Integration Test / Integration tests GitHub Actions: Unit Tests / Run Unit Test ✅ 5 other checks have passed 5/8 successful checks

Release - flytesnacks v0.3.179 New release published by flyte-bot Changelog • 3148c6c Add mlflow example (#960) flyteorg/flytesnacks

Release - flytesnacks v0.3.180 New release published by flyte-bot Changelog • 343072b Upgrading aws sdk to use WebIdentityTokenCredentialsProvider (#967) flyteorg/flytesnacks

1 new commit pushed to

<https://github.com/flyteorg/flyteadmin/tree/master|master>

by katrogan

<https://github.com/flyteorg/flyteadmin/commit/188b982aa6d64cf29683fe91a343d85e1c27564a|188b982a>

- Don't truncate error messages when listing task executions (#536) flyteorg/flyteadmin

Release - v1.1.78 New release published by flyte-bot Changelog • 188b982 Don't truncate error messages when listing task executions (#536) flyteorg/flyteadmin

#1543 Use fsspec placeholder for structured dataset Pull request opened by wild-endeavor TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit GitHub Actions: build-plugins (3.8, flytekit-hive) GitHub Actions: build-plugins (3.8, flytekit-greatexpectations) GitHub Actions: build-plugins (3.8, flytekit-duckdb) GitHub Actions: build-plugins (3.8, flytekit-dolt) GitHub Actions: build-plugins (3.8, flytekit-deck-standard) GitHub Actions: build-plugins (3.8, flytekit-dbt) GitHub Actions: build-plugins (3.8, flytekit-data-fsspec) GitHub Actions: build-plugins (3.8, flytekit-dask) GitHub Actions: build-plugins (3.8, flytekit-bigquery) GitHub Actions: build-plugins (3.8, flytekit-aws-sagemaker) GitHub Actions: build-plugins (3.8, flytekit-aws-batch) GitHub Actions: build-plugins (3.8, flytekit-aws-athena) GitHub Actions: docs GitHub Actions: build (windows-latest, 3.9) GitHub Actions: build (windows-latest, 3.8) GitHub Actions: build (ubuntu-latest, 3.10) GitHub Actions: build (ubuntu-latest, 3.9) GitHub Actions: build (ubuntu-latest, 3.8) GitHub Actions: lint ✅ 2 other checks have passed 2/21 successful checks

#716 chore: show correct app version in info Pull request opened by ursucarina TL;DR Display correct app version in info. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Reading app version from correct source. Final outcome: flyteorg/flyteconsole GitHub Actions: Build & Push FlyteConsole Image GitHub Actions: Get Release Tag GitHub Actions: Generate Release ✅ 6 other checks have passed 6/9 successful checks

#1535 [Core feature] Convert List[Any] to a single pickle file Pull request opened by Yicheng-Lu-llll TL;DR Currently, flyte create N (size of the list) pickle files if the output type is List[Any]. This slows down serialization. it takes more than 15 mins to upload the pickles to s3 if the size of the list is 1000 (see Slack Thread) This PR: • converts the entire list into a single pickle file by default. • takes batches of list items and uploads each batch as a separate pickle file if the user annotates the batch size(see below).

@task
def t1() -> Annotated[List[Any],  100]

Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Originally, in

ListTransformer.to_literal

, Items are converted one by one. It is time-consuming, especially for large lists. flytekit/flytekit/core/type_engine.py Line 977 in </flyteorg/flytekit/commit/8ccb5dde044f9dce3e216a8cea58dd464fb3ab6b|8ccb5dd> This PR checks if python_type is list[flytePickle]. If so, batch the list of items and call

FlytePickleTransformer.to_literal

. Measure the performance Test file:

from typing import List, Dict, Any
from flytekit import task, Resources, workflow

@task(
    limits=Resources(mem="4Gi",cpu="1"),
    disable_deck=True,
)
def test() -> List[Any]:
    return  [{"a": {0: "foo"}}] * 10000

@workflow
def wf():
    test()

if __name__ == "__main__":
    wf()

# yichenglu/flytekit:mytest5 contains this PR
# pyflyte run  --image yichenglu/flytekit:mytest5 --remote ./test.py  wf
# <http://cr.flyte.org/flyteorg/flytekit:py3.10-1.4.0b1|cr.flyte.org/flyteorg/flytekit:py3.10-1.4.0b1> is the default image
# pyflyte run  --image <http://cr.flyte.org/flyteorg/flytekit:py3.10-1.4.0b1|cr.flyte.org/flyteorg/flytekit:py3.10-1.4.0b1> --remote ./test.py  wf

In an 4VCPU, 16G RAM EC2 instance, measuring the performance in flyte cluster using the above test file and the commands： • Prior to these changes, , converting a list with type

List[Any]

with 10000 items needs 157 seconds. • After this PR, convertinga list with type

List[Dict[str, Any]]

with 10000 items only needs 7 seconds. backward compatibility: • case1: old task's cached output -> new task

from typing import List, Dict, Any, Annotated
from flytekit import task, Resources, workflow
@task(
    limits=Resources(mem="4Gi",cpu="1"),
    disable_deck=True,
    container_image="{{.image.oldImage}}",
    cache=True, cache_version="1.0"   
)
def task0() -> List[List[Any]]:
    return   [ ["foo","foo","foo"] ] * 2
@task(
    limits=Resources(mem="4Gi",cpu="1"),
    disable_deck=True,
    container_image="{{.image.prImage}}",
)
def task1(data: List[List[Any]]) -> List[List[Any]]:
    print(data)
    return  data
@workflow
def wf():
    data = task0()
    task1(data=data)
# pyflyte run --image prImage="yichenglu/flytekit:mytest4" --image oldImage="<http://cr.flyte.org/flyteorg/flytekit:py3.10-1.4.1|cr.flyte.org/flyteorg/flytekit:py3.10-1.4.1>" --remote ./test.py  wf

Backward compatibility is preserved in this case. Can see pod log ton double check. • case2: new task(image include this PR) -> old task(image without this PR)

from typing import List, Dict, Any, Annotated
from flytekit import task, Resources, workflow
@task(
    limits=Resources(mem="4Gi",cpu="1"),
    disable_deck=True,
    container_image="{{.image.prImage}}",
)
def task0() -> List[List[Any]]:
    return   [ ["foo","foo","foo"] ] * 2
@task(
    limits=Resources(mem="4Gi",cpu="1"),
    disable_deck=True,
    container_image="{{.image.oldImage}}"
)
def task1(data: List[List[Any]]) -> List[List[Any]]:
    print(data)
    return  data
@workflow
def wf():
    data = task0()
    task1(data=data)
# pyflyte run --image prImage="yichenglu/flytekit:mytest4" --image oldImage="<http://cr.flyte.org/flyteorg/flytekit:py3.10-1.4.1|cr.flyte.org/flyteorg/flytekit:py3.10-1.4.1>" --remote ./test.py  wf

[[['foo', 'foo', 'foo']], [['foo', 'foo', 'foo']]]

will be printed if see the pod's log This occurs because in Task0, the outputs are batched, while in Task 1, which does not include this PR, the results are not expanded. While slightly impacting backward compatibility, it is justifiable to ask users to update their images for two primary reasons: • Before the implementation of this PR, users faced lengthy wait times for tasks involving pickling to be completed. • Only tasks with the type list[any] will be affected. In most scenarios, users can still utilize the old images or cached outputs. Tracking Issue flyteorg/flyte#3207 Follow-up issue NA flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

#381 Init customTokenSource.refreshTime Pull request opened by andrewwdye TL;DR Follow up to #377. This change sets

refreshTime

based on the cached token expiry when instantiating

customTokenSource

. Otherwise

refreshTime

defaults to 0, effectively nullifying the cache we just added since it's always viewed as refresh window exceeded on first use. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description This change • Chooses a new

refreshTime

if a cached token is available, otherwise default to empty time • Moves token source provider tests to dedicated file • Adds new test

TestCustomTokenSource_GetTokenSource

• Removes overly verbose logging Tracking Issue NA Follow-up issue NA flyteorg/flyteidl GitHub Actions: Generate Swagger Code ✅ 11 other checks have passed 11/12 successful checks

#1544 Make `FlyteFile` compatible with `Annotated[..., HashMethod]` Pull request opened by AdrianoKF See issue #3424 TL;DR This PR makes `FlyteFile`s usable with offloaded hash calculation via

Annotated[FlyteFile, HashMethod(...)]

, so that they can be cached based on file contents, e.g. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description In

FlyteFilePathTransformer.to_literal()

,

Annotated

type aliases are now handled by unwrapping the first type argument to the alias, before checking if they are a subclass of

FlyteFile

. Tracking Issue flyteorg/flyte#3424 Follow-up issue NA flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

#717 FE: Update flyteconsole to Node 18 Pull request opened by james-union flyteorg/flyte#3415 Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description With node v16, semantic-release did not work Tracking Issue flyteorg/flyte#3415 flyteorg/flyteconsole GitHub Actions: unit_tests_with_coverage GitHub Actions: lint_project GitHub Actions: extract_branch ✅ 2 other checks have passed 2/5 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by ursucarina

<https://github.com/flyteorg/flyteconsole/commit/0a6ad81411f744ee682e58e3a9d51baca9e64e00|0a6ad814>

- FE: Update flyteconsole to Node 18 (#717) flyteorg/flyteconsole

#718 fix: left nav doesn't accurately update on workflow version page Pull request opened by ursucarina TL;DR Fixes left nav logic that doesn't accurately update when navigating to a specific workflow version Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Updated the workflow detail path to include the workflow version path To repro: • checkout branch, run & navigate to <https//localhost.development.uniondemo.run3000/console/projects/flytesnacks/domains/development/workflow/1.wf/version/DW6W2hJD0NJeK_EOj5SmPw==|this link >Result: The workflows left nav item should be highlighted • compare to the same link in production flyteorg/flyteconsole GitHub Actions: Build & Push FlyteConsole Image GitHub Actions: Get Release Tag GitHub Actions: Generate Release GitHub Actions: Build Flyteconsole Image / Push to Github Registry GitHub Actions: lint_project GitHub Actions: unit_tests_with_coverage ✅ 3 other checks have passed 3/9 successful checks

#3430 Differentiate between cache disabled and cache put failure Issue created by hamersaw A user has a cache put failure which we can confirm is set in the admin DB as the correct code. However, when it is displayed in the UI, the icon is the same as the "cache disabled" icon and hovering displays as such. It would be useful if there were visual differentiation between these two status'. flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by ursucarina

<https://github.com/flyteorg/flyteconsole/commit/172664a79ca49055acb1dbdd376c01b777604f27|172664a7>

- chore: allow complex workflow names (#715) flyteorg/flyteconsole

#3431 Add project filter like flytectl --filter.xxx flags Issue created by kimjunil Our team have many projects. so I want to filter the project by label on the main screen like "flytectl get project --filter.fieldSelector" command. flyteorg/flyte

#3432 Bug: Archived launchplans are not currently being filtered in the launch plan list Issue created by jsonporter Users have requested support for archiving launch plans. Acceptance Criteria: • Users can archive LP from search UI • Archived LPs are excluded from LP search UI • Filter text is 'Include Archived' • Filter is inclusive (like task/workflow toggle implemented in 661). • Archived LP are not included in drop-down on launch form UI unless that LP is the current selected LP (eg, relaunching an execution) flyteorg/flyte

#3434 Feat: Add support to archive launch plans Issue created by jsonporter Users have requested the ability to archive a launch plan. Note: We will need to investigate the best way to handle how this feature will coincide with things like Relaunch flows. Acceptance Criteria: • User can archive/unarchive LP from launch plan search • Solution for relaunch flows when dealing with archived launch plans flyteorg/flyte

#644 Mapped Tasks not showing cache status correctly. Issue created by jsonporter Users are noticing some inconsistent behavior in regards to the cache status for mapped tasks. It seems like cache values are only out of sync in running/unknown phases (and eventually rectify once the parent task is

succeeded

). This ticket will track root causing and fixing the issue. flyteorg/flyteconsole

#3435 Distinguish between task and workflow executions in the UI Issue created by katrogan From #3048 flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by ursucarina

<https://github.com/flyteorg/flyteconsole/commit/a299398e9e2a7180a845d116f27cbdb7db46bae2|a299398e>

- chore: show correct app version in info (#716) flyteorg/flyteconsole

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by ursucarina

<https://github.com/flyteorg/flyteconsole/commit/ffb161cb2266094f86ff416bc52ec57c56ef7a80|ffb161cb>

- fix: left nav doesn't accurately update on workflow version page (#718) flyteorg/flyteconsole

Release - v1.4.8 New release published by flyte-bot 1.4.8 (2023-03-09) Bug Fixes • left nav doesn't accurately update on workflow version page (#718) (ffb161c) flyteorg/flyteconsole