#3527 [Core feature] [Performance] Using top level pickle objects vs nested pickle objects Issue created by kumare3 Motivation: Why do you think this is important? A user can unknowingly end up in a performance issue, when using top level pickle objects vs nested pickled objects inside outer containers like lists. This is because on observing a collection time, Flytekit will try to use

list

transformer and then recursively find a pickle transformer. Also due the default pickle transport is a File, a list of even simple objects can result in 100's of files and since they are asynchronously written can cause a huge performance penalty. Example

import attr
import flytekit


## Worflow below is slow
@attr.define
class Foo:
    x: int


@flytekit.task
def t1_slow() -> list[Foo]:
    return [Foo(i) for i in range(2000)]


@flytekit.task
def t2_slow(foos: list[Foo]) -> list[int]:
    return [foo.x for foo in foos]


@flytekit.workflow
def wf_slow() -> list[int]:
    foos = t1_slow()
    return t2_slow(foos=foos)


## Worflow below is fast
@attr.define
class Container:
    foos: list[Foo]


@flytekit.task
def t1_fast() -> Container:
    return Container([Foo(i) for i in range(2000)])

@flytekit.task
def t2_fast(container: Container) -> list[int]:
    return [foo.x for foo in container.foos]

@flytekit.workflow
def wf_fast() -> list[int]:
    container = t1_fast()
    return t2_fast(container=container)

For context, wf_slow can be around 30x slower in some random cases. Goal: What should the final outcome look like, ideally? In both the cases the performance should be close to possible. For pickle objects in general that are small, we could use a binary transport? Describe alternatives you've considered NA Propose: Link/Inline OR Additional context In pickle transformer we can add the following snippet https://github.com/flyteorg/flytekit/blob/de2878903b594069f1629d7ee11bf59ab04b59f2/flytekit/types/pickle/pickle.py#L51

if lv.scalar.blob:
   ... # continue with current logic
elif lv.scalar.binary:
   ... # load bytearray

and while

to_literal

x = cloudpickle.dumps()
if len(x) < threshold:
   lv.scalar.binary = x
else
 continue

Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by jeevb

<https://github.com/flyteorg/flyte/commit/27ae9e0b206575ead777619acc515835f4858fed|27ae9e0b>

- Add support for updating flytecopilot image in flyte-binary helm chart (#3526) flyteorg/flyte

#1562 TypeTransformer for TensorFlow model Pull request opened by samhita-alla TL;DR This PR enables

tf.keras.Model

to be utilized as a Flyte type. Type ☐ Bug Fix ☐ Feature ☑︎ Plugin Are all requirements met? ☑︎ Code completed ☐ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue flyteorg/flyte#2570 Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit GitHub Actions: build-plugins (3.8, flytekit-onnx-pytorch) GitHub Actions: build-plugins (3.8, flytekit-modin) GitHub Actions: build-plugins (3.8, flytekit-kf-tensorflow) GitHub Actions: build-plugins (3.8, flytekit-kf-pytorch) GitHub Actions: build-plugins (3.8, flytekit-kf-mpi) GitHub Actions: build-plugins (3.8, flytekit-k8s-pod) GitHub Actions: build-plugins (3.8, flytekit-hive) GitHub Actions: build-plugins (3.8, flytekit-greatexpectations) GitHub Actions: build-plugins (3.8, flytekit-duckdb) GitHub Actions: build-plugins (3.8, flytekit-dolt) GitHub Actions: build-plugins (3.8, flytekit-deck-standard) GitHub Actions: build-plugins (3.8, flytekit-dbt) GitHub Actions: build-plugins (3.8, flytekit-data-fsspec) GitHub Actions: build-plugins (3.8, flytekit-dask) GitHub Actions: build-plugins (3.8, flytekit-bigquery) GitHub Actions: build-plugins (3.8, flytekit-aws-sagemaker) GitHub Actions: build-plugins (3.8, flytekit-aws-batch) GitHub Actions: build-plugins (3.8, flytekit-aws-athena) GitHub Actions: docs GitHub Actions: build (windows-latest, 3.9) GitHub Actions: build (windows-latest, 3.8) GitHub Actions: build (ubuntu-latest, 3.10) GitHub Actions: build (ubuntu-latest, 3.9) GitHub Actions: build (ubuntu-latest, 3.8) GitHub Actions: Docs Warnings GitHub Actions: lint ✅ 2 other checks have passed 2/28 successful checks

#1241 feat: add TypeTransformer for Tensorflow Model Pull request opened by techytushar TL;DR This PR adds support for: 1. TypeTransformer for Tensorflow Model type Note: I was not able to run the

pip-compile

commands to generate the new

requirements

files, it failed on

protobuf

version error Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue flyteorg/flyte#2570 Follow-up issue NA flyteorg/flytekit ✅ All checks have passed 2/2 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by andresgomezfrr

<https://github.com/flyteorg/flytekit-java/commit/abcf4db44feeb41b175e66bfe22c6e66e36ba4c9|abcf4db4>

- Add support to apply node using a Map[String, SdkBindingData[_]] (#216) flyteorg/flytekit-java

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by github-actions[bot]

<https://github.com/flyteorg/flytekit-java/commit/dea408199370f3d3eb170b98eaeb547e815600ed|dea40819>

- [maven-release-plugin] prepare release 0.4.13 flyteorg/flytekit-java

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by github-actions[bot]

<https://github.com/flyteorg/flytekit-java/commit/970b2f58f856b702a206a6b03533c213d0ce46a1|970b2f58>

- [maven-release-plugin] prepare for next development iteration flyteorg/flytekit-java

#213 Fail on missing mocks Pull request opened by bergman on <!date^1679328654^{date_short}|2023-03-20T16:10:54Z> Read then delete this section - Make sure to use a concise title for the pull-request. _- Use #patch, #minor or #major in the pull-request title to bump the corresponding version. Otherwise, the patch version will be bumped. More details_ TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit-java GitHub Actions: build-snapshot DCO: DCO ✅ 1 other check has passed 1/3 successful checks

#1563 [v1.3 Backport] Less strict docker dependency versions (#1536) Pull request opened by jcugat Backport #1536 to v1.3 branch flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

#1564 [v1.2 Backport] Less strict docker dependency versions (#1536) Pull request opened by jcugat Backport #1536 to v1.2 branch flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytepropeller/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flytepropeller/commit/e4624120c85c0c3ebf5a93f75ae57065b306e4a0|e4624120>

- Add reported_at timestamps to events (#529) flyteorg/flytepropeller

Release - v1.1.72 New release published by flyte-bot Changelog • e462412 Add reported_at timestamps to events (#529) flyteorg/flytepropeller

#731 chore: add build:watch to all packages Pull request opened by ursucarina TL;DR Dev process improvement Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Adds a build:watch script for all packages that also publishes to the yalc store when a file changes flyteorg/flyteconsole Codecov: 66.64% (-0.12%) compared to b062639 Codecov: 0.00% of diff hit (target 66.76%) GitHub Actions: Build & Push FlyteConsole Image GitHub Actions: Get Release Tag GitHub Actions: Generate Release ✅ 6 other checks have passed 6/11 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytepropeller/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flytepropeller/commit/b1e54821c98093f75e4ab19a9545d3b704ac6d99|b1e54821>

- Including all upstream node deps on BranchNode subnode execution (#543) flyteorg/flytepropeller

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flytekit/commit/36fe5813d83477f91056f61bd234e9dd232ec4a2|36fe5813>

- add pod_template and pod_template_name arguments for ContainerTask (#1515) flyteorg/flytekit

#422 [UI] Properly handle loading/error cases in Launch form Issue created by schottra The Launch form was refactored to use a state machine and does not currently have any logic to show a loading indicator in cases where data takes more than 1 second to load. It also will not show any errors when data fails to load (other than logging to the browser console). This issue if for handling those two cases in the following scenarios: • Loading Task/Workflow versions for the selectors • Loading Launch Plans after selecting a workflow version • Loading inputs after selecting a Task Version or WorkflowVersion/Launch Plan combination Additionally, we may want to consider some logic for when we fail to load search results in the Workflow/Task version input. Some guidelines: • We have been slowly replacing circular progress indicators with

<Skeleton />

components where appropriate. This is probably one of those cases, as we don't want a succession of various loading indicators flashing into different spots in the form. • Loading indicators should only show if requests take longer than 1 second. • Errors should use a small or medium variant of the

DataError

component. flyteorg/flyteconsole

#3528 [UI] Convert remaining data fetching operations to use `react-query` Issue created by schottra Most of the fetching logic for the Execution details and Task Execution Details pages was converted to use

react-query

instead of our custom fetching hook. While it's not hurting anything to continue using the old fetching logic for other parts of the application, maintaining two different patterns creates confusion and more cognitive load for contributors. This task tracks the complete deprecation and removal of

useFetchableData

in favor of

react-query

. Flyte component ☐ Overall ☐ Flyte Setup and Installation scripts ☐ Flyte Documentation ☐ Flyte communication (slack/email etc) ☐ FlytePropeller ☐ FlyteIDL (Flyte specification language) ☐ Flytekit (Python SDK) ☐ FlyteAdmin (Control Plane service) ☐ FlytePlugins ☐ DataCatalog ☐ FlyteStdlib (common libraries) ☑︎ FlyteConsole (UI) ☐ Other flyteorg/flyte

#3529 [UI] Show more information for Branch nodes Issue created by schottra The UI will currently only show minimal information about a branch node in the list of NodeExecutions. This could be enhanced to show things such as the if/else condition and maybe which branch was taken at runtime. Flyte component ☑︎ FlyteConsole (UI) flyteorg/flyte

#3530 [Feature] More visibility into when the latest registered workflow was added Issue created by kosigz-lyft Motivation: Why do you think this is important? The "latest" version of a workflow in the UI doesn't have a date associated with it, so it's hard to tell "is this the workflow I just registered?" Goal: What should the final outcome look like, ideally? Having more data about when an update occurred in-line can give higher confidence that it's the workflow I just registered. Describe alternatives you've considered N/A Flyte component ☐ Overall ☐ Flyte Setup and Installation scripts ☐ Flyte Documentation ☐ Flyte communication (slack/email etc) ☐ FlytePropeller ☐ FlyteIDL (Flyte specification language) ☐ Flytekit (Python SDK) ☐ FlyteAdmin (Control Plane service) ☐ FlytePlugins ☐ DataCatalog ☐ FlyteStdlib (common libraries) ☑︎ FlyteConsole (UI) ☐ Other [Optional] Propose: Link/Inline Perhaps consider adding date/time to each version. Additional context Is this a blocker for you to adopt Flyte N/A flyteorg/flyte

#443 [Feature] composite tasks not visible in workflow UI page before running the workflow Issue created by kosigz-lyft Motivation: Why do you think this is important? While using the Flyte UI, I found that you cannot see the tasks within a workflow without running the workflow. This can be confusing in the case where a developer adds a new task, but is not sure if it's been picked up correctly before running the workflow. Goal: What should the final outcome look like, ideally? Ideally, when one opens the Flyte console, they should be able to see the tasks that are part of a workflow without running the workflow. Currently, all you can see are runs and you have to have a started run to see the actual execution graph. Describe alternatives you've considered The workaround involves running the workflow to see the tasks. Flyte component ☐ Overall ☐ Flyte Setup and Installation scripts ☐ Flyte Documentation ☐ Flyte communication (slack/email etc) ☐ FlytePropeller ☐ FlyteIDL (Flyte specification language) ☑︎ Flytekit (Python SDK) ☐ FlyteAdmin (Control Plane service) ☐ FlytePlugins ☐ DataCatalog ☐ FlyteStdlib (common libraries) ☑︎ FlyteConsole (UI) ☐ Other Additional context n/a Is this a blocker for you to adopt Flyte n/a already using Flyte. flyteorg/flyteconsole

#438 [Feature]UX for versions of a Workflow, Launch Plan and Task Issue created by kumare3 Motivation: Why do you think this is important? When a user registers a new version of a workflow, the current UX makes it extremely easy to launch an execution, but, it makes it hard to • discover: find workflows, tasks, launch plans and their versions as a chronology • debug: Check if a specific version was registered • Collate: executions collected by a specific version - might help the user finding which executions they are looking for • Document: Documentation - like code - is versioned and it might be useful to show the latest, but allow users to dive into previous versions Goal: What should the final outcome look like, ideally? UX should make it possible to find all registered versions of a workflow/task/launchplan, look for a specific SHA, look at the details of the configuration etc in the UI. User should be able to group all executions by a version User should be able to view the documentation of a workflow, launchplan and task Describe alternatives you've considered NA Flyte component ☐ Overall ☐ Flyte Setup and Installation scripts ☐ Flyte Documentation ☐ Flyte communication (slack/email etc) ☐ FlytePropeller ☐ FlyteIDL (Flyte specification language) ☐ Flytekit (Python SDK) ☐ FlyteAdmin (Control Plane service) ☐ FlytePlugins ☐ DataCatalog ☐ FlyteStdlib (common libraries) ☑︎ FlyteConsole (UI) ☐ Other Comments This is lower priority than other things, but, will definitely help in creating a better debugging experience in Flyte. flyteorg/flyteconsole

#3531 [Feature]Create Project flow in the UI Issue created by kumare3 Motivation: Why do you think this is important? When starting off with a Flyte installation, today it comes set with some default projects. Ideally it should come with no projects (or projects with examples.) Goal: What should the final outcome look like, ideally? The UI should have a flow to create new projects. Describe alternatives you've considered Use the CLI and this is the current option. Flytectl would be another option. Thus this is not a high priority item, but would improve the first launch workflow for the user. Flyte component ☐ Overall ☐ Flyte Setup and Installation scripts ☐ Flyte Documentation ☐ Flyte communication (slack/email etc) ☐ FlytePropeller ☐ FlyteIDL (Flyte specification language) ☐ Flytekit (Python SDK) ☐ FlyteAdmin (Control Plane service) ☐ FlytePlugins ☐ DataCatalog ☐ FlyteStdlib (common libraries) ☑︎ FlyteConsole (UI) ☐ Other [Optional] Propose: Link/Inline the UI storyboard should be created Is this a blocker for you to adopt Flyte NA, but helps improve startup flyteorg/flyte

#408 [BUG] Unable to scroll page body with Details panel open and a large error message expanded. Issue created by schottra 1. View a failed execution with a large top-level error message. 2. Click on any node execution name to open the details panel 3. Expand the top-level error 4. Attempt to scroll down on the page. Expected: Scrolling should respond as normal Actual: Unable to scroll the page until the Details panel is closed. flyteorg/flyteconsole

#407 [Feature]FlyteConsole LaunchForm should show show LaunchPlan Spec details Issue created by kumare3 Motivation: Why do you think this is important? LaunchPlan spec has many other details besides just Inputs. There is no way to browse all launch plans. We should be able to review other details of a launchplan - for e.g. • RAW path • labels • annotations etc Eventually user should be able to update them too for an execution Goal: What should the final outcome look like, ideally? Show in the UI details of a launchplan Describe alternatives you've considered FlyteCLI is the only option. Flyte component ☐ Overall ☐ Flyte Setup and Installation scripts ☐ Flyte Documentation ☐ Flyte communication (slack/email etc) ☐ FlytePropeller ☐ FlyteIDL (Flyte specification language) ☐ Flytekit (Python SDK) ☐ FlyteAdmin (Control Plane service) ☐ FlytePlugins ☐ DataCatalog ☐ FlyteStdlib (common libraries) ☑︎ FlyteConsole (UI) ☐ Other Additional context It makes it extremely hard for users to understand their settings for a launchplan Is this a blocker for you to adopt Flyte NA flyteorg/flyteconsole

#405 Launch form needs some copy to explain how inputs work Issue created by schottra There is no clear delineation between the selection of a Workflow version / LaunchPlan and the section for entering inputs. It would also help to have a small amount of copy explaining how to use the form and how default values are applied. Screenshot below is a potential solution: flyteorg/flyteconsole

#3532 Allow editing of descriptions via the UI Issue created by schottra Projects, Workflows, Launch Plans, and Tasks can all have descriptions associated with them. There are endpoints that allow setting these values. But currently there is no UX for doing so in the Console. A basic implementation would be the click-to-edit pencil which saves on blur. Before editing (pencil icon) Edit state with cancel/save buttons Display of save errors goes below the buttons flyteorg/flyte

Release - v1.1.73 New release published by flyte-bot Changelog • b1e5482 Including all upstream node deps on BranchNode subnode execution (#543) flyteorg/flytepropeller

#3533 [Core feature] Support no task execution in a conditional branch Issue created by yubofredwang Motivation: Why do you think this is important? As a Flyte user, I often have tasks that only need to run under certain conditions, and if conditions not satisfied, no task needs to be executed. Currently, the conditional in Flyte does not support skipping one of the branch without any task execution. This means that we will need to use a

noop

task to skip the branch. This will lead to waste of time during execution. For example:

@task
def noop():
     ...

conditional("generate_artifact")
.if_(need_to_generate.is_true())
.then(generate_artifact())
.else_(noop())

This makes the code ugly and UI messy. We will need to figure out a way to cleanly handle this. Goal: What should the final outcome look like, ideally? Ideally, we can introduce something like`end()` or

skip()

operation in conditional to skip the branch, similar to

fail()

operation here: https://github.com/flyteorg/flytekit/blob/master/flytekit/core/condition.py#L295 Describe alternatives you've considered noop() task with cache is being used right now. it is bad coding practice and confusing in the UI. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#544 [Snyk] Security upgrade alpine from 3.13.7 to 3.15 Pull request opened by snyk-bot Keeping your Docker base image up-to-date means you’ll benefit from security fixes in the latest version of your chosen image. Changes included in this PR • boilerplate/flyte/golang_dockerfile/Dockerfile.GoTemplate We recommend upgrading to

alpine:3.15

, as this image has only 1 known vulnerabilities. To do this, merge this pull request, then verify your application still works as expected. Some of the most important vulnerabilities in your base image include: * * * Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs. For more information: https://camo.githubusercontent.com/b2f3d3f1a44598de62b980a6dae389b2a137e646ac357e0eb6db13387aeed1dc/68747470733a2f2f6170692e7365676d656e742e696f2f76312f706978656c2f747261636b3f646174613d65794a33636d6c305a55746c65534936496e4a79576d785a634564485932527954485a7362306c596430645563566734576b4652546e4e434f5545774969776959573576626e6c746233567a535751694f694a6c597a497a4d6d49304e79303259574a694c5452694f4463744f5467354f4331685a6a4d345a6d5933595745304e6d59694c434a6c646d567564434936496c425349485a705a58646c5a434973496e42796233426c636e52705a584d694f6e736963484a4a5a434936496d566a4d6a4d79596a51334c545a68596d49744e4749344e7930354f446b344c57466d4d7a686d5a6a6468595451325a694a3966513d3d 🧐 View latest project report 🛠️ Adjust project settings * * * Learn how to fix vulnerabilities with free interactive lessons: 🦉 Learn about vulnerability in an interactive lesson of Snyk Learn. flyteorg/flyteadmin DCO: DCO ✅ 1 other check has passed 1/2 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by ursucarina

<https://github.com/flyteorg/flyteconsole/commit/6816a5f9e8c8b3844f1084496553cd6f65290280|6816a5f9>

- chore: add build:watch to all packages (#731) flyteorg/flyteconsole