GitHub
04/07/2023, 6:19 PMGitHub
04/07/2023, 7:12 PMSKIPPED
due to an upstream node failure, these tasks are not retried, and the execution is incorrectly marked as SUCCEEDED
anyway.
Expected behavior
SKIPPED
tasks should also be retried on recovery if the upstream node succeeds.
Additional context to reproduce
import random
from flytekit import task, workflow
from flytekit.core.workflow import WorkflowFailurePolicy
@task
def pass_through(input1: int) -> int:
return input1
@task
def fail(input1: int) -> int:
if random.randint(0, 10) < 7:
assert False
return input1
@workflow(failure_policy=WorkflowFailurePolicy.FAIL_AFTER_EXECUTABLE_NODES_COMPLETE)
def wf(wf_input: int) -> tuple[int, int]:
a = fail(input1=wf_input)
b = pass_through(input1=wf_input)
c = pass_through(input1=a)
return b, c
Execute against a flyte-sandbox like so:
pyflyte run --remote --image <http://cr.flyte.org/flyteorg/flytekit:py3.10-latest|cr.flyte.org/flyteorg/flytekit:py3.10-latest> test/recovery.py wf --wf_input 3
Since the task failure is non-deterministic, keep retrying until the first node fails. The last node should now be marked as SKIPPED
. Then, recover until the first node succeeds and observe the behavior.
Screenshots
Initial failure:
Screenshot 2023-04-07 at 11 57 41 AM▾
Screenshot 2023-04-07 at 11 58 02 AM▾
Screenshot 2023-04-07 at 12 03 29 PM▾
GitHub
04/07/2023, 7:43 PMGitHub
04/07/2023, 8:07 PM<https://github.com/flyteorg/flyteidl/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flyteidl/commit/7b19eb3c4cf3dbfe02c4d5c5f4fdd1ee0c6551bc|7b19eb3c>
- Rust pb (#392)
flyteorg/flyteidlGitHub
04/07/2023, 8:08 PMGitHub
04/07/2023, 8:24 PM<https://github.com/flyteorg/flyte/tree/master|master>
by hamersaw
<https://github.com/flyteorg/flyte/commit/db9162c65712d5e716c751e7e9f73a536107c969|db9162c6>
- Start controller-runtime manager in single binary (#3547)
flyteorg/flyteGitHub
04/07/2023, 8:26 PMGitHub
04/07/2023, 8:34 PM<https://github.com/flyteorg/flyteplugins/tree/master|master>
by hamersaw
<https://github.com/flyteorg/flyteplugins/commit/4f02c2ca8d461538e43a1cd6250bcfd23cd448c4|4f02c2ca>
- deprecated TransitionTypeBarrier (#336)
flyteorg/flytepluginsGitHub
04/07/2023, 8:41 PMGitHub
04/07/2023, 9:29 PMattemptRecovery
. This surfaced an issue where nodes in the SKIPPED
phase were reported as SKIPPED
during recovery when they should be executed in the recovered workflow.
This PR correctly reports the phase of previously SKIPPED
nodes as UNDEFINED
to ensure the nodes are correctly evaluated during workflow recoveries.
Type
☑︎ Bug Fix
☐ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☑︎ Smoke tested
☐ Unit tests added
☑︎ Code documentation added
☐ Any pending items have an associated Issue
Complete description
^^^
Tracking Issue
fixes flyteorg/flyte#3578
Follow-up issue
NA
flyteorg/flytepropeller
GitHub Actions: Goreleaser
GitHub Actions: Build & Push Flytepropeller Image
GitHub Actions: Bump Version
✅ 11 other checks have passed
11/14 successful checksGitHub
04/07/2023, 9:42 PM<https://github.com/flyteorg/helm-flyte-core/tree/main|main>
by wild-endeavor
<https://github.com/flyteorg/helm-flyte-core/commit/70e390b0871925b7e6a59d5fb6331e0b6436bd2a|70e390b0>
- Initial commit
flyteorg/helm-flyte-coreGitHub
04/07/2023, 10:11 PM<https://github.com/flyteorg/flytesnacks/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flytesnacks/commit/3819bc23336dad12d1ba7eddc946775096ed0335|3819bc23>
- Remove note about python versions from getting started (#982)
flyteorg/flytesnacksGitHub
04/07/2023, 10:12 PM<https://github.com/flyteorg/flytepropeller/tree/master|master>
by hamersaw
<https://github.com/flyteorg/flytepropeller/commit/7abdcd77b606739bafc780a2bd45196c5bff908f|7abdcd77>
- returning 'Undefined' when recovering a SKIPPED node so that the node is executed (#551)
flyteorg/flytepropellerGitHub
04/07/2023, 10:49 PMGitHub
04/08/2023, 12:07 AMflyte-core
and flyte-binary
) to ghcr on every commit to master. This makes it easier for people testing early versions of Flyte to pick up and test changes.
This should not affect the existing charts published under https://flyteorg.github.io/flyte at all.
Links:
https://github.com/orgs/flyteorg/packages/container/package/helm-charts%2Fflyte-binary
https://github.com/orgs/flyteorg/packages/container/package/helm-charts%2Fflyte-core
flyteorg/flyte
✅ All checks have passed
6/6 successful checksGitHub
04/08/2023, 1:10 AMGitHub
04/08/2023, 9:39 AMimage▾
GitHub
04/08/2023, 4:44 PM<https://github.com/flyteorg/flytestdlib/tree/master|master>
by EngHabu
<https://github.com/flyteorg/flytestdlib/commit/3b737b5f309826fe1ba13709221b1a31aa42a18b|3b737b5f>
- Add a new requestID to Log fields (#157)
flyteorg/flytestdlibGitHub
04/08/2023, 4:53 PMGitHub
04/09/2023, 8:02 PMGitHub
04/10/2023, 1:17 AMGitHub
04/10/2023, 2:17 AMGitHub
04/10/2023, 2:51 PMdask cuda worker
i.e. that dask-cuda is present on the specified image.
Type
☑︎ Feature
☑︎ Plugin
Are all requirements met?
☐ Code completed
☐ Smoke tested
☑︎ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
Currently we build an arg to submit to the container and inject values such as threads and memory from the limits specified in the resources object in the dask plugin object. Dask cuda gives us many built in gpu tools and all that is required is to ensure dask cuda is installed and then to start out workers with dask-cuda-worker
. So, in order to accomodate, we simply check for gpu limits in the dask plugin resources and overwrite the first arg to dask-cuda-worker
.
flyteorg/flyteplugins
✅ All checks have passed
6/6 successful checksGitHub
04/10/2023, 4:00 PMGitHub
04/10/2023, 5:01 PM<https://github.com/flyteorg/flytesnacks/tree/master|master>
by jeevb
<https://github.com/flyteorg/flytesnacks/commit/ccc945229e383a80c3183668ae4af73e68749de1|ccc94522>
- Read minio connection parameters from environment variable if available (#983)
flyteorg/flytesnacksGitHub
04/10/2023, 5:35 PM<https://github.com/flyteorg/flyte/tree/master|master>
by wild-endeavor
<https://github.com/flyteorg/flyte/commit/feb8d60a0a228de2494043976440ad77a50ac90c|feb8d60a>
- Github actions pipeline for publishing developmental helm charts (#3579)
flyteorg/flyteGitHub
04/10/2023, 5:40 PMGitHub
04/10/2023, 6:13 PMGitHub
04/10/2023, 6:34 PMGitHub
04/10/2023, 6:42 PM<https://github.com/flyteorg/flytekit-python-template/tree/main|main>
by zeryx
<https://github.com/flyteorg/flytekit-python-template/commit/fb66dc78fd6aed54247878febaa0f23808195c92|fb66dc78>
- Add getting started template (#25)
flyteorg/flytekit-python-template