#3547 Start controller-runtime manager in single binary Pull request opened by hamersaw Tracking issue fixes #3546 Describe your changes Starting the controller-runtime manager if either propeller or the propeller webhook is not disabled. Check all the applicable boxes ☑︎ I updated the documentation accordingly. ☑︎ All new and existing tests passed. ☑︎ All commits are signed-off. Screenshots NA Note to reviewers NA flyteorg/flyte ✅ All checks have passed 9/9 successful checks

#3578 [BUG] Recovery fails to retry tasks that were `SKIPPED` in previous execution due to an upstream node failure in Flyte >=1.4.x Issue created by jeevb Describe the bug When recovering an execution with tasks that were

SKIPPED

due to an upstream node failure, these tasks are not retried, and the execution is incorrectly marked as

SUCCEEDED

anyway. Expected behavior

SKIPPED

tasks should also be retried on recovery if the upstream node succeeds. Additional context to reproduce

import random

from flytekit import task, workflow
from flytekit.core.workflow import WorkflowFailurePolicy


@task
def pass_through(input1: int) -> int:
    return input1


@task
def fail(input1: int) -> int:
    if random.randint(0, 10) < 7:
        assert False
    return input1


@workflow(failure_policy=WorkflowFailurePolicy.FAIL_AFTER_EXECUTABLE_NODES_COMPLETE)
def wf(wf_input: int) -> tuple[int, int]:
    a = fail(input1=wf_input)
    b = pass_through(input1=wf_input)
    c = pass_through(input1=a)
    return b, c

Execute against a flyte-sandbox like so:

pyflyte run --remote --image <http://cr.flyte.org/flyteorg/flytekit:py3.10-latest|cr.flyte.org/flyteorg/flytekit:py3.10-latest> test/recovery.py wf --wf_input 3

Since the task failure is non-deterministic, keep retrying until the first node fails. The last node should now be marked as

SKIPPED

. Then, recover until the first node succeeds and observe the behavior. Screenshots Initial failure: After "successful" recovery: How it SHOULD work (behavior in Flyte v1.3.0): Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#393 Migrate to buf remote plugins v1 Pull request opened by eapolinario TL;DR Move to buf remote plugins (and away from soon-to-be-deprecated alpha remote generation) Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Buf remote plugins replace the alpha remote code generation as described in https://buf.build/blog/remote-packages-remote-plugins-approaching-v1/. The deadline is the end of April, so getting slightly ahead. I also bumped the version of some the plugins (and regenerated the stubs). Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA flyteorg/flyteidl ✅ All checks have passed 13/13 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteidl/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flyteidl/commit/7b19eb3c4cf3dbfe02c4d5c5f4fdd1ee0c6551bc|7b19eb3c>

- Rust pb (#392) flyteorg/flyteidl

Release - v1.3.17 New release published by flyte-bot Changelog • 7b19eb3 Rust pb (#392) flyteorg/flyteidl

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flyte/commit/db9162c65712d5e716c751e7e9f73a536107c969|db9162c6>

- Start controller-runtime manager in single binary (#3547) flyteorg/flyte

#982 Remove note about python versions from getting started Pull request opened by eapolinario flytekit 1.5.0 is out, which removes this version check. flyteorg/flytesnacks ✅ All checks have passed 3/3 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteplugins/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flyteplugins/commit/4f02c2ca8d461538e43a1cd6250bcfd23cd448c4|4f02c2ca>

- deprecated TransitionTypeBarrier (#336) flyteorg/flyteplugins

Release - v1.0.48 New release published by flyte-bot Changelog • 4f02c2c deprecated TransitionTypeBarrier (#336) flyteorg/flyteplugins

#551 Fixing recovering of SKIPPED nodes Pull request opened by hamersaw TL;DR The PR to enable recovering dynamic workflows when they have already compiled the workflow closure updated the node executor to always use the phase returned from

attemptRecovery

. This surfaced an issue where nodes in the

SKIPPED

phase were reported as

SKIPPED

during recovery when they should be executed in the recovered workflow. This PR correctly reports the phase of previously

SKIPPED

nodes as

UNDEFINED

to ensure the nodes are correctly evaluated during workflow recoveries. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☑︎ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue fixes flyteorg/flyte#3578 Follow-up issue NA flyteorg/flytepropeller GitHub Actions: Goreleaser GitHub Actions: Build & Push Flytepropeller Image GitHub Actions: Bump Version ✅ 11 other checks have passed 11/14 successful checks

1 new commit pushed to

<https://github.com/flyteorg/helm-flyte-core/tree/main|main>

by wild-endeavor

<https://github.com/flyteorg/helm-flyte-core/commit/70e390b0871925b7e6a59d5fb6331e0b6436bd2a|70e390b0>

- Initial commit flyteorg/helm-flyte-core

1 new commit pushed to

<https://github.com/flyteorg/flytesnacks/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flytesnacks/commit/3819bc23336dad12d1ba7eddc946775096ed0335|3819bc23>

- Remove note about python versions from getting started (#982) flyteorg/flytesnacks

1 new commit pushed to

<https://github.com/flyteorg/flytepropeller/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flytepropeller/commit/7abdcd77b606739bafc780a2bd45196c5bff908f|7abdcd77>

- returning 'Undefined' when recovering a SKIPPED node so that the node is executed (#551) flyteorg/flytepropeller

Release - v1.1.78 New release published by flyte-bot Changelog • 7abdcd7 returning 'Undefined' when recovering a SKIPPED node so that the node is executed (#551) flyteorg/flytepropeller

#3579 github actions pipeline for publishing developmental helm charts Pull request opened by jeevb This publishes the two Helm charts supported by Flyte (

flyte-core

and

flyte-binary

) to ghcr on every commit to master. This makes it easier for people testing early versions of Flyte to pick up and test changes. This should not affect the existing charts published under https://flyteorg.github.io/flyte at all. Links: https://github.com/orgs/flyteorg/packages/container/package/helm-charts%2Fflyte-binary https://github.com/orgs/flyteorg/packages/container/package/helm-charts%2Fflyte-core flyteorg/flyte ✅ All checks have passed 6/6 successful checks

#3580 [refactor] [flytekit] Map task improvements Issue created by wild-endeavor Map tasks in flytekit currently are expressed as a MapPythonTask, a separate object. The special things we currently do for map tasks • create a separate object • special local execute behavior • special casing in entrypoint.py • special casing in translator.py The goal for this ticket is to attempt to get rid of as much special casing in flytekit that we have. Ultimately when a map task run, it's the underlying task that runs, just with a different input/output location, for the most part. We should be able to remove a substantial amount of the special logic we have for map tasks in flytekit. It may be possible for us to one day get to a place where we don't have to register a separate row in the tasks table for the map task, but that is not the goal of this ticket. The reason is caching - we want users to be able to cache the results of the map task at the map task layer, not just the underlying tasks. This ticket may or may not be related to the array node work. flyteorg/flyte

#1587 Add support nested FlyteFile in pyflyte run Pull request opened by pingsutw TL;DR Fixes https://flyte-org.slack.com/archives/CP2HDHKE1/p1680845682408899 Add support nested FlyteFile in pyflyte run. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Tracking Issue https://flyte-org.slack.com/archives/CP2HDHKE1/p1680845682408899 Follow-up issue NA flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytestdlib/tree/master|master>

by EngHabu

<https://github.com/flyteorg/flytestdlib/commit/3b737b5f309826fe1ba13709221b1a31aa42a18b|3b737b5f>

- Add a new requestID to Log fields (#157) flyteorg/flytestdlib

Release - v1.0.17 New release published by github-actions[bot] Changelog • 3b737b5 Add a new requestID to Log fields (#157) flyteorg/flytestdlib

#401 Update workflow_execution_config Pull request opened by pingsutw TL;DR Update the format of workflow-execution-config Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue NA Follow-up issue NA flyteorg/flytectl GitHub Actions: Generate documentation GitHub Actions: Goreleaser GitHub Actions: Bump Version ✅ 9 other checks have passed 9/12 successful checks

#983 Read minio connection parameters from environment variable if available Pull request opened by jeevb flyteorg/flytesnacks GitHub Actions: Build & Push to GHCR (cookbook/case_studies/ml_training/mnist_classifier) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/ml_training/house_price_prediction) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/feature_engineering/feast_integration) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/feature_engineering/eda) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/bioinformatics/blast) GitHub Actions: Docs Warnings ✅ 3 other checks have passed 3/9 successful checks

#984 Doc for using raw container in the map task Pull request opened by pingsutw • When using rawContainer in the map task, Input will become string representation of a list, so we have to convert it to a list. • Users also need to get the array index first to process specific element in the list. flyteorg/flytesnacks GitHub Actions: Build & Push to GHCR (cookbook/integrations/flytekit_plugins/sql) GitHub Actions: Build & Push to GHCR (cookbook/integrations/flytekit_plugins/papermilltasks) GitHub Actions: Build & Push to GHCR (cookbook/integrations/flytekit_plugins/pandera_examples) GitHub Actions: Build & Push to GHCR (cookbook/integrations/flytekit_plugins/onnx_examples) GitHub Actions: Build & Push to GHCR (cookbook/integrations/flytekit_plugins/modin_examples) GitHub Actions: Build & Push to GHCR (cookbook/integrations/flytekit_plugins/mlflow_example) GitHub Actions: Build & Push to GHCR (cookbook/integrations/flytekit_plugins/greatexpectations) GitHub Actions: Build & Push to GHCR (cookbook/integrations/flytekit_plugins/duckdb_examples) GitHub Actions: Build & Push to GHCR (cookbook/integrations/flytekit_plugins/dolt) GitHub Actions: Build & Push to GHCR (cookbook/integrations/flytekit_plugins/dbt_example) GitHub Actions: Build & Push to GHCR (cookbook/integrations/external_services/databricks) GitHub Actions: Build & Push to GHCR (cookbook/integrations/aws/sagemaker_training) GitHub Actions: Build & Push to GHCR (cookbook/integrations/aws/sagemaker_pytorch) GitHub Actions: Build & Push to GHCR (cookbook/integrations/aws/batch) GitHub Actions: Build & Push to GHCR (cookbook/integrations/aws/athena) GitHub Actions: Build & Push to GHCR (cookbook/core) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/ml_training/spark_horovod) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/ml_training/pima_diabetes) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/ml_training/nlp_processing) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/ml_training/mnist_classifier) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/ml_training/house_price_prediction) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/feature_engineering/feast_integration) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/feature_engineering/eda) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/bioinformatics/blast) GitHub Actions: Docs Warnings ✅ 3 other checks have passed 3/28 successful checks

#339 Use dask cuda worker when GPU resource &gt; 0 Pull request opened by ljstrnadiii on <!date^1680493347^{date_short}|2023-04-03T03:42:27Z> TL;DR If a gpu has been specified in the dask plugin resource, assume we want to use

dask cuda worker

i.e. that dask-cuda is present on the specified image. Type ☑︎ Feature ☑︎ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Currently we build an arg to submit to the container and inject values such as threads and memory from the limits specified in the resources object in the dask plugin object. Dask cuda gives us many built in gpu tools and all that is required is to ensure dask cuda is installed and then to start out workers with

dask-cuda-worker

. So, in order to accomodate, we simply check for gpu limits in the dask plugin resources and overwrite the first arg to

dask-cuda-worker

. flyteorg/flyteplugins ✅ All checks have passed 6/6 successful checks

Pending review on flyteorg/flyteidl - Manage reminder [#297] Proposal : Settings proto (pmahindrakar-oss) 3 months stale · a year old · Waiting on _ @Haytham Abuelfutuh, katrogan_

1 new commit pushed to

<https://github.com/flyteorg/flytesnacks/tree/master|master>

by jeevb

<https://github.com/flyteorg/flytesnacks/commit/ccc945229e383a80c3183668ae4af73e68749de1|ccc94522>

- Read minio connection parameters from environment variable if available (#983) flyteorg/flytesnacks

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by wild-endeavor

<https://github.com/flyteorg/flyte/commit/feb8d60a0a228de2494043976440ad77a50ac90c|feb8d60a>

- Github actions pipeline for publishing developmental helm charts (#3579) flyteorg/flyte

Release - flytesnacks v0.3.191 New release published by flyte-bot Changelog • ccc9452 Read minio connection parameters from environment variable if available (#983) • 3819bc2 Remove note about python versions from getting started (#982) flyteorg/flytesnacks

#3581 Run functional tests against sandbox-bundled Pull request opened by jeevb flyteorg/flyte GitHub Actions: build-and-push-sandbox-bundled-image GitHub Actions: sandbox-bundled-functional-tests ✅ 8 other checks have passed 8/10 successful checks

flyteorg/helm-flyte-core Repository deleted by wild-endeavor flyteorg/helm-flyte-core

1 new commit pushed to

<https://github.com/flyteorg/flytekit-python-template/tree/main|main>

by zeryx

<https://github.com/flyteorg/flytekit-python-template/commit/fb66dc78fd6aed54247878febaa0f23808195c92|fb66dc78>

- Add getting started template (#25) flyteorg/flytekit-python-template