Release - v1.0.30 New release published by flyte-bot Changelog • b8d3ea9 fix comments (#309) flyteorg/flyteplugins

#3247 [Core feature] Realtime applications on Flyte Issue created by davidmirror-ops Motivation: Why do you think this is important? Acccording to @dylanwilder: "we have an use case where the overhead of spinning up flyte components provides some lag in our user experience, but we can't duplicate business logic into a web backend" Goal: What should the final outcome look like, ideally? "we have user interactive loops for which we currently wait for long processing calculations ("what if" scenario modeling). some of these models take a while to run and a long feedback loop is expected. others are quick (think mathematical formulas) and expectation is realtime feedback on parameter changes. so not the typical streaming pipeline application but we could definitely leverage it. the major bottleneck in some tasks is spool times, and in others we have use cases where we could run caculations for smaller sets of data for faster output but spool time would quickly become the bottleneck here as well" Describe alternatives you've considered TBD Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by narape

<https://github.com/flyteorg/flytekit-java/commit/ebc38e24bf9004d86d9bcb897974e88ebcdd4e97|ebc38e24>

- Make Build run on all branches (#172) flyteorg/flytekit-java

#514 Add include_archived flag and unit tests for it Pull request opened by pmahindrakar-oss on <!date^1674142716^{date_short}|2023-01-19T15:38:36Z> Signed-off-by: pmahindrakar-oss prafulla.mahindrakar@gmail.com TL;DR Current behavior of ListExecutions API is to return all executions except for archived executions if no state filter is passed Inorder to keep the old behavior the same, adding a new includeArchived as part of the ListResourceRquest If includeArchived is passed then no state filter override is added. This behavior current only exists for executions and not available on launchplans, tasks and workflows as they dont support archival. Followup with adding this behavior for other entities including saving the state flyteorg/flyteidl#361 Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flyteadmin GitHub Actions: Goreleaser GitHub Actions: Build & Push Flyteadmin Image GitHub Actions: Build & Push Flytescheduler Image GitHub Actions: Bump Version Codecov: 25.00% of diff hit (target 60.04%) GitHub Actions: End2End Test / End to End tests GitHub Actions: Lint / Run Lint ✅ 7 other checks have passed 7/14 successful checks

#361 Add include archived Pull request opened by pmahindrakar-oss on <!date^1674142728^{date_short}|2023-01-19T15:38:48Z> Read then delete this section - Make sure to use a concise title for the pull-request. _- Use #patch, #minor or #major in the pull-request title to bump the corresponding version. Otherwise, the patch version will be bumped. More details_ TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flyteidl GitHub Actions: Generate Swagger Code ✅ 11 other checks have passed 11/12 successful checks

#518 Copy dynamicJobSpec from partially completed DynamicTask on recovery Pull request opened by hamersaw DO NOT MERGE waiting on PRs: flyteorg/flyteidl#360 flyteorg/flyteadmin#513 TL;DR Since node IDs in dynamic job specs are not computed deterministically between executions recovering partially completed dynamic tasks can result in data corruptions. In this PR we attempt to recover the

DynamicJobSpec

from previous the previous

DynamicTask

execution during node recovery. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue fixes flyteorg/flyte#3121 Follow-up issue NA flyteorg/flytepropeller GitHub Actions: Goreleaser GitHub Actions: Build & Push Flytepropeller Image GitHub Actions: Bump Version ✅ 11 other checks have passed 11/14 successful checks

#672 Include Archived workflows Pull request opened by james-union (#661) Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Sending [Archived, NonArchived] state with

VALUE_IN

filter operation to get all executions/tasks Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ flyteorg/flyte#661 flyteorg/flyteconsole DCO: DCO ✅ 1 other check has passed 1/2 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by ursucarina

<https://github.com/flyteorg/flyteconsole/commit/924bb7e2fdab3c306de32a03924cf01a09f1c050|924bb7e2>

- chore: Refactor flyteconsole (#654) flyteorg/flyteconsole

#653 Refactor flyteconsole Issue created by ursucarina flyteorg/flyteconsole

#654 chore: Refactor flyteconsole Pull request opened by ursucarina TL;DR Added support for custom console components to be injected into flyteconsole without needing to host the code in the flyteconsole codebase. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Changed the flyteconsole codebase to separate the main console into ./packages that can be imported into custom implementations. The flyteconsole app will be deployed from ./website. For contributing to the codebase, all changes should be made to ./packages/console. Tracking Issue fixes #653 flyteorg/flyteconsole GitHub Actions: Build & Push FlyteConsole Image GitHub Actions: Get Release Tag GitHub Actions: Generate Release Codecov: 67.62% (-0.66%) compared to 516bbff ✅ 7 other checks have passed 7/11 successful checks

#673 chore: update flyteidl version @1.3.4 Pull request opened by ursucarina TL;DR Update @flyteorg/flyteidl to version 1.3.4 Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Update @flyteorg/flyteidl to version 1.3.4 flyteorg/flyteconsole Codecov: 67.62% (-0.11%) compared to 924bb7e GitHub Actions: Build Flyteconsole Image / Push to Github Registry ✅ 6 other checks have passed 6/8 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by ursucarina

<https://github.com/flyteorg/flyteconsole/commit/ab91dfc09a072aaa1eac7063b675d4ff92800c15|ab91dfc0>

- chore: update flyteidl version @1.3.4 (#673) flyteorg/flyteconsole

1 new commit pushed to

<https://github.com/flyteorg/flytepropeller/tree/master|master>

by pmahindrakar-oss

<https://github.com/flyteorg/flytepropeller/commit/20146304738db96e8a15cf9a824233676a7560ea|20146304>

- Update flyteidl version (#490) flyteorg/flytepropeller

#490 Update flyteidl version Pull request opened by flyte-bot Update flyteidl version • Auto-generated by [flyte-bot] flyteorg/flytepropeller GitHub Actions: Goreleaser GitHub Actions: Build & Push Flytepropeller Image GitHub Actions: Bump Version ✅ 11 other checks have passed 11/14 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by wild-endeavor

<https://github.com/flyteorg/flytekit/commit/5dd887cf5cbe817a35d682a32886f304a97fc910|5dd887cf>

- Flip the settings for channel and logger (#1415) flyteorg/flytekit

#2959 [Core feature] Add Audience parameter in ClientSecret Authentication flow. Issue created by shahwar9 Motivation: Why do you think this is important? We are deploying Flyte as its ML Workflow orchestrator. Auth0 server expects that authentication flow uses

Audience

in its handshake process. When using Flyte with Authentication enabled, Flyte Scheduler and Flyte Propeller talk to Auth0 server with ClientSecret flow. Having no option to set Audience parameter in ClientSecret flow has halted our deployment as we are not able to use Flyte with Authentication. Read more in Audience: https://auth0.com/docs/secure/tokens/access-tokens/get-management-api-access-tokens-for-production#get-access-tokens The reason this is important is it expands the use case of Flyte with users taking extra precaution by using Audience. It does not change anything for the users who do not want to use Audience, so it is a non-breaking change to make. Goal: What should the final outcome look like, ideally? Flytepropeller, Flyte Scheduler and any other backend service that uses ClientSecret flow for authentication should have the option to enable Audience via values.yaml configs. An example for setting Audience within config is already present at Flyteadmin config: flyte/charts/flyte-core/values-keycloak-idp-flyteclients-without-browser.yaml Line 527 in </flyteorg/flyte/commit/01f7cd5c5fa7b713062fc8338821a956422639ba|01f7cd5> Describe alternatives you've considered I am not sure of an alternative solution in this case, the company policies are pretty tight to use Audience in every Authentication flow. Propose: Link/Inline OR Additional context Since ClientSecret flow is using Auth2 golang SDK so I dug a little bit into it and probably something that can help Flyte implement is this: golang/oauth2@01b79d9 Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#515 Update idl to latest Pull request opened by pmahindrakar-oss Signed-off-by: pmahindrakar-oss prafulla.mahindrakar@gmail.com Read then delete this section - Make sure to use a concise title for the pull-request. _- Use #patch, #minor or #major in the pull-request title to bump the corresponding version. Otherwise, the patch version will be bumped. More details_ TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flyteadmin GitHub Actions: Build & Push Flytescheduler Image GitHub Actions: Build & Push Flyteadmin Image GitHub Actions: Goreleaser GitHub Actions: Bump Version ✅ 10 other checks have passed 10/14 successful checks

#310 Transition to failed state if the task cannot be created Pull request opened by pingsutw Fixes flyteorg/flyte#3281 TL;DR Transition to failed state if the task isn't created. otherwise, the task will run forever. 1 round (phase:

PhaseNotStarted

) • Webapi plugin try to allocate token 2 round (state:

PhaseAllocationTokenAcquired

, phase:

queued

): • Plugin fail to create a task, and will not update the task state. The state is

PhaseAllocationTokenAcquired

in this round • Task handler send the task event (queued) to admin • However we transition the phase from queued to running here • And we will record the node event (running) in the node executor 3 round (state:

PhaseAllocationTokenAcquired

, phase:

running

): • Will try to create a task, and fail again • In the screenshot below, you can see the node phase is

running

, but the task state is

queued

4+ round • Repeat 3 round Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Before After Tracking Issue flyteorg/flyte#3281 It's related to this issue propeller panic because taskCtx.ResourceMeta() is nil. However, it should never happen. it was happening because we tried to abort the task that doesn't exist. Follow-up issue NA flyteorg/flyteplugins ✅ All checks have passed 6/6 successful checks

#173 Fix merge errors Pull request opened by andresgomezfrr Read then delete this section - Make sure to use a concise title for the pull-request. _- Use #patch, #minor or #major in the pull-request title to bump the corresponding version. Otherwise, the patch version will be bumped. More details_ TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit-java ✅ All checks have passed 3/3 successful checks

#519 Transition to failed state if the k8s resource cannot be created Pull request opened by pingsutw TL;DR Related to flyteorg/flyteplugins#310 Mark task state as failed if the k8s resource cannot be created. otherwise, the propeller will keep trying to create CR and task will run forever. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue Follow-up issue NA flyteorg/flytepropeller ✅ All checks have passed 2/2 successful checks

#3249 [Docs] Fix up caching docs as per user feedback Issue created by ppiegaze Description In this Slack thread we find the following advice (we should act on it): Niels Bantilan @Sidharth (Sid) yes our flyte.org website revamp (coming soon!) should feature caching a lot more prominently. Out of curiosity, do you find the current docs on caching clear and understandable? Sidharth(Sid) 18 hours ago I feel it has everything a person needs to understand Caching, but it needs a thorough read, maybe even a few times for a complete new comer to workflow tools. Also, terms related to caching such as "task Signature" could be simpler in my opinion. For example "Any changes to task Function". Prefect V1 docs for caching - https://docs-v1.prefect.io/core/concepts/persistence.html#output-caching-based-on-a-file-target Its pretty old, but has similar concepts explained in simple sentences. Im just sharing it so that I can show you what kind of understanding fits a crowd who are new to these tools. But I do wish the above prefect page to be more technical. Closer to flyte's way of explaining. Basically a middle ground will be nice. Also in Flyte current stable docs, I would wish for more details on "local cache storage" and "remote cache storage". The above prefect v1 doc gives some insight to it. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3250 [flytekit-java] Change SdkBindingData to be typed Issue created by andresgomezfrr Motivation: Why do you think this is important? After evaluating the current development experience using flytekit-java with some of our users. We've detected some improvements that could help to do the development experience more smooth. One of them is the difficulty that the users have to know what is the specific type of SdkBindingData when working with remote tasks. Currently, the user needs to jump into the remote code to check the input/output classes to know what is the specific type of each attribute. It could be very tedious because you need to find the repository of this remote task or jump using the IDE and check the input. Goal: What should the final outcome look like, ideally? The final outcome is that we want to change the

SdkBindingData

to a typed version

SdkBindingData<?>

. This new version allows the user to know the inner type during the normal development and also helps to ensure that you are setting the current input to a task because if not you get a type mismatch error at compile time. Describe alternatives you've considered no alternatives Propose: Link/Inline OR Additional context flyteorg/flytekit-java#164 Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3251 [flytekit-java] Add typed transform outputs Issue created by andresgomezfrr Motivation: Why do you think this is important? Right now the user needs to know the output of a task because the only way to get it is using the

getOutput("<attributeName>")

. This is difficult during the development because you need to check the output class or docs to know what are the available types, also writing the name manually produces typos errors that only appear at registration time. The proposal here is to have some way to use the output class during the workflow development on that way you can access directly to the output class attributes. Goal: What should the final outcome look like, ideally? SumTaskOutput class

@AutoValue
  public abstract static class SumOutput {

    public abstract SdkBindingData<Long> c();

    public static SumOutput create(SdkBindingData<Long> c) {
      return new AutoValue_SumTask_SumOutput(c);
    }
  }

Current status:

SdkBindingData fib3 = builder.apply("fib-3", SumTask.of(fib1, fib2)).getOutput("c");

Desire status:

SdkBindingData fib3 = builder.apply("fib-3", SumTask.of(fib1, fib2)).getOutputs().c();

This allows the user to auto-discover the output attributes using an IDE, and also. check that you choose only available attributes at compile time. Describe alternatives you've considered no alternatives Propose: Link/Inline OR Additional context flyteorg/flytekit-java#164 Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3252 [Housekeeping] [flytekit-java] Add support to struct using the typed SdkBindingData and outputs Issue created by andresgomezfrr Describe the issue Add support to structs using the typed SdkBindingData and also the output type. What if we do not do this? If we don't do this the struct is not supported as input/output using java/scala.is Related component(s) flytekit-java Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by narape

<https://github.com/flyteorg/flytekit-java/commit/cd1026bda94387184864c6969b644c8be8df12dc|cd1026bd>

- Change SdkBindingData to be typed and add typed transform outputs #minor (#164) flyteorg/flytekit-java

#3253 [Housekeeping] Any plans to more actively maintain bazelrules_flyte? Issue created by samjeckert Describe the issue Hi all! We are very excited about Flyte as a potential replacement to an inhouse tool we've developed, but as we are a monorepo/bazel shop we are a bit daunted with getting the tool integrated. The bazelrules_flyte repo highlights a lot of the things we might want to do ourselves, but given its state its unlikely we can use it directly. What if we do not do this? We likely end up in a less than ideal state with Flyte until if/when we can dedicate the time to deep dive and develop our own rules_flyte equivalent. Related component(s) No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1416 [Docs] SynchronousFlyteClient API reference #3095 Pull request opened by ppiegaze TL;DR Fix SynchronousFlyteClient API reference #3095 Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Docs completed ☑︎ Locally tested Complete description Added new 'Clients' section in Flytekit API docs that pulls in docstring from

flytekit.clients.friendly.SynchronousFlyteClient

and

flytekit.clients.raw.RawSynchronousFlyteClient

Tracking Issue flyteorg/flyte#3095 Follow-up issue NA flyteorg/flytekit GitHub Actions: build-plugins (3.8, flytekit-pandera) GitHub Actions: build-plugins (3.8, flytekit-onnx-scikitlearn) GitHub Actions: build-plugins (3.8, flytekit-onnx-pytorch) GitHub Actions: build-plugins (3.8, flytekit-modin) GitHub Actions: build-plugins (3.8, flytekit-kf-tensorflow) GitHub Actions: build-plugins (3.8, flytekit-kf-pytorch) GitHub Actions: build-plugins (3.8, flytekit-kf-mpi) GitHub Actions: build-plugins (3.8, flytekit-k8s-pod) GitHub Actions: build-plugins (3.8, flytekit-hive) GitHub Actions: build-plugins (3.8, flytekit-greatexpectations) GitHub Actions: build-plugins (3.8, flytekit-dolt) GitHub Actions: build-plugins (3.8, flytekit-deck-standard) GitHub Actions: build-plugins (3.8, flytekit-dbt) GitHub Actions: build-plugins (3.8, flytekit-data-fsspec) GitHub Actions: build-plugins (3.8, flytekit-dask) GitHub Actions: build-plugins (3.8, flytekit-bigquery) GitHub Actions: build-plugins (3.8, flytekit-aws-sagemaker) GitHub Actions: build-plugins (3.8, flytekit-aws-batch) GitHub Actions: build-plugins (3.8, flytekit-aws-athena) GitHub Actions: build (windows-latest, 3.9) GitHub Actions: build (windows-latest, 3.8) GitHub Actions: build (windows-latest, 3.7, -spark2) GitHub Actions: build (windows-latest, 3.7) GitHub Actions: build (ubuntu-latest, 3.10) GitHub Actions: build (ubuntu-latest, 3.9) GitHub Actions: build (ubuntu-latest, 3.8) GitHub Actions: build (ubuntu-latest, 3.7, -spark2) GitHub Actions: build (ubuntu-latest, 3.7) GitHub Actions: lint ✅ 1 other check has passed 1/30 successful checks

#3254 [BUG] copilot uploader sidecar is started even if no outputs are defined Issue created by flixr Describe the bug I have ContainerTasks without any outputs, but it still creates the copilot uploader sidecar, which then exits immediately marking the pod as not ready... There is a check in copilot plugin but it doesn't cover this case properly, since outputs is set to empty map in flytekit core interface So in the copilot plugin it should not just check for nil (None), but also for empty... Expected behavior copilot uploader sidecar is not started Additional context to reproduce simple example:

from flytekit import ContainerTask, kwtypes, workflow

hello_task = ContainerTask(
    name="hello",
    image="ubuntu:20.04",
    inputs=kwtypes(name=str),
    command=["echo", "hello {{.inputs.name}}"]
)

@workflow
def say_hello(name: str):
    hello_task(name=name)

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3255 [Housekeeping] Create community repo Issue created by davidmirror-ops Describe the issue Currently, community related documentation (basically the Code of Conduct and a link to the contributing docs in the site) resides in the

flyte

repo. In order to centralize the community-related information like Governance, communication methods, Code of Conduct, contribution guides, among others, a flyteorg/community repo would be useful. Also it would help with issue tracking, avoiding having community-related issues stored alongside a number of issues related to the codebase. The overarching goal is to make easier for new and established community members to discover and track any progress on making the Flyte community even more welcoming for users and contributors. Also, if there would be additional OSS projects in the future hosted under flyteorg (even in a monorepo), /community could still be the place to centralize all the procedures for the flyteorg community What if we do not do this? With the use of labels we could filter out the community-related issues, but we would have to create new labels for this to happen. If someone new to the project explores current issues, it would probably not be evident that community-related discussions/ideas are also located there Related component(s) No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by kumare3

<https://github.com/flyteorg/flytekit/commit/310aea3296170126433f3e7b2949c6a6ea1203c0|310aea32>

- Preserving Exception in the LazyEntity fetch (#1412) flyteorg/flytekit