GitHub
11/30/2023, 7:00 PMGitHub
11/30/2023, 7:24 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flytekit/commit/b12c84796d7365ffb1b4dce579e35d20906d00f6|b12c8479>
- Fix default paramaters when launching launchplans from within other workflows (#1972)
flyteorg/flytekitGitHub
11/30/2023, 8:00 PMWorkflowPhaseHandlingFailureNode
if workflow fails and workflow has a failureNode.
• Transition subworkflow status to running
if subworkflow has a failureNode and subworkflow fails.
Example
Three workflows
1. wf1: both workflow and subworkflow have a failure node. propeller will create two tasks when failing
2. wf2: Only one workflow has a failure node. propeller will create one task when failing
3. wf3: failure node is a workflowNode. propeller will create another workflow when failing
flytectl demo start --dev
make compile # compile the single binary
flyte start --config flyte-single-binary-local.yaml
pip install git+<https://github.com/flyteorg/flytekit.git@2582a8bc16f80ab3a7101af8360e9d7212236e43>
pyflyte run --remote workflow.py wf
import typing
from click.testing import CliRunner
from flytekit import task, workflow, ImageSpec, WorkflowFailurePolicy
from flytekit.clis.sdk_in_container import pyflyte
from flytekit.types.error.error import FlyteError
new_flytekit = "git+<https://github.com/flyteorg/flytekit.git@5a415107b0aff272a16eb147860a65d47a10c4d8>"
image_spec = ImageSpec(packages=[new_flytekit], apt_packages=["git"], registry="pingsutw")
@task(container_image=image_spec)
def create_cluster(name: str):
print(f"Creating cluster: {name}")
@task(container_image=image_spec)
def t1(a: int, b: str):
print(f"{a} {b}")
raise ValueError("Fail!")
@task(container_image=image_spec)
def delete_cluster(name: str, err: typing.Optional[FlyteError] = None):
print(f"Deleting cluster {name}")
print(err)
@task(container_image=image_spec)
def clean_up(name: str, err: typing.Optional[FlyteError] = None):
print(f"Deleting cluster {name} due to {err}")
print(err)
@workflow(on_failure=clean_up, failure_policy=WorkflowFailurePolicy.FAIL_AFTER_EXECUTABLE_NODES_COMPLETE)
def subwf(name: str = "kevin"):
c = create_cluster(name=name)
t = t1(a=1, b="2")
d = delete_cluster(name=name)
c >> t >> d
@workflow(on_failure=clean_up, failure_policy=WorkflowFailurePolicy.FAIL_AFTER_EXECUTABLE_NODES_COMPLETE)
def wf1(name: str = "kevin"):
c = create_cluster(name=name)
subwf(name="pingsutw")
t = t1(a=1, b="2")
d = delete_cluster(name=name)
c >> t >> d
@workflow(on_failure=clean_up, failure_policy=WorkflowFailurePolicy.FAIL_AFTER_EXECUTABLE_NODES_COMPLETE)
def wf2(name: str = "kevin"):
c = create_cluster(name=name)
t = t1(a=1, b="2")
d = delete_cluster(name=name)
c >> t >> d
@workflow
def clean_up_wf(name: str = "kevin"):
return create_cluster(name=name)
@workflow(on_failure=clean_up_wf, failure_policy=WorkflowFailurePolicy.FAIL_AFTER_EXECUTABLE_NODES_COMPLETE)
def wf3(name: str = "Kevin"):
c = create_cluster(name=name)
t = t1(a=1, b="2")
d = delete_cluster(name=name)
c >> t >> d
if __name__ == '__main__':
runner = CliRunner()
result = runner.invoke(pyflyte.main, ["run", "--remote", "failure_node.py", "wf2"])
print(result.output)
Tracking issue
#1506
Screenshots
image
flyteorg/flyte
Codecov: 59.45% (-0.02%) compared to a8f4904
✅ 29 other checks have passed
29/30 successful checksGitHub
11/30/2023, 8:30 PM<https://github.com/flyteorg/flyte/tree/master|master>
by jeevb
<https://github.com/flyteorg/flyte/commit/7481b3d0adc508fe07c5e2aa624621687efd41c4|7481b3d0>
- Correctly handle resource overrides in KF plugins (#4467)
flyteorg/flyteGitHub
11/30/2023, 10:59 PM{"json":{"exec_id":"f3d082a54f9b94988ba4","node":"workflowsexamplesecrettask","ns":"flytesnacks-development","res_ver":"108245851","routine":"worker-3","src":"controller.go:36","tasktype":"python-task","wf":"flytesnacks:development:.flytegen.workflows.example.secret_task"},"level":"info","msg":"The back-off handler for [/v1, Kind=pod,flytesnacks-development] has been loaded.\n","ts":"2023-11-30T21:26:53Z"}
2023/11/30 21:26:53 http: panic serving 10.2.149.237:48966: runtime error: invalid memory address or nil pointer dereference
goroutine 26608 [running]:
net/http.(*conn).serve.func1()
/usr/local/go/src/net/http/server.go:1850 +0xbf
panic({0x2f4dc00, 0x6164a90})
/usr/local/go/src/runtime/panic.go:890 +0x262
<http://sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Decoder).DecodeRaw(0x0|sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Decoder).DecodeRaw(0x0>, {{0xc005ce8000, 0x1b70, 0x2000}, {0x0, 0x0}}, {0x46ca258, 0xc0037b0000})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/webhook/admission/decode.go:77 +0x15a
<http://sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Decoder).Decode(_|sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Decoder).Decode(_>, {{{0xc00377fcb0, 0x24}, {{0x0, 0x0}, {0xc005fd30c0, 0x2}, {0xc005fd30c2, 0x3}}, {{0x0, ...}, ...}, ...}}, ...)
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/webhook/admission/decode.go:49 +0x8e
<http://github.com/flyteorg/flyte/flytepropeller/pkg/webhook.(*PodMutator).Handle(_|github.com/flyteorg/flyte/flytepropeller/pkg/webhook.(*PodMutator).Handle(_>, {_, _}, {{{0xc00377fcb0, 0x24}, {{0x0, 0x0}, {0xc005fd30c0, 0x2}, {0xc005fd30c2, ...}}, ...}})
/flyteorg/build/flytepropeller/pkg/webhook/pod.go:86 +0xc8
<http://sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle(_|sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle(_>, {_, _}, {{{0xc00377fcb0, 0x24}, {{0x0, 0x0}, {0xc005fd30c0, 0x2}, {0xc005fd30c2, ...}}, ...}})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/webhook/admission/webhook.go:169 +0x20b
<http://sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP(0xc008872a50|sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP(0xc008872a50>, {0x7f1ba49b1e40?, 0xc00a05f450}, 0xc005cc0100)
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/webhook/admission/http.go:98 +0xc94
<http://github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1({0x7f1ba49b1e40|github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1({0x7f1ba49b1e40>, 0xc00a05f450}, 0x46e7500?)
/go/pkg/mod/github.com/prometheus/client_golang@v1.16.0/prometheus/promhttp/instrument_server.go:60 +0xd4
net/http.HandlerFunc.ServeHTTP(0x46e7560?, {0x7f1ba49b1e40?, 0xc00a05f450?}, 0x7e1286?)
/usr/local/go/src/net/http/server.go:2109 +0x2f
<http://github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x46e7560|github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x46e7560>?, 0xc00175bce0?}, 0xc005cc0100)
/go/pkg/mod/github.com/prometheus/client_golang@v1.16.0/prometheus/promhttp/instrument_server.go:147 +0xc5
net/http.HandlerFunc.ServeHTTP(0x7e0ce5?, {0x46e7560?, 0xc00175bce0?}, 0x40?)
/usr/local/go/src/net/http/server.go:2109 +0x2f
<http://github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2(|github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2(>{0x46e7560, 0xc00175bce0}, 0xc005cc0100)
/go/pkg/mod/github.com/prometheus/client_golang@v1.16.0/prometheus/promhttp/instrument_server.go:109 +0xc7
net/http.HandlerFunc.ServeHTTP(0xc00175bce0?, {0x46e7560?, 0xc00175bce0?}, 0x34ec45f?)
/usr/local/go/src/net/http/server.go:2109 +0x2f
net/http.(*ServeMux).ServeHTTP(0xc00377fc35?, {0x46e7560, 0xc00175bce0}, 0xc005cc0100)
/usr/local/go/src/net/http/server.go:2487 +0x149
net/http.serverHandler.ServeHTTP({0x46cec18?}, {0x46e7560, 0xc00175bce0}, 0xc005cc0100)
/usr/local/go/src/net/http/server.go:2947 +0x30c
net/http.(*conn).serve(0xc000c6a8c0, {0x46e9780, 0xc008ab12f0})
/usr/local/go/src/net/http/server.go:1991 +0x607
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:3102 +0x4db
We recently bumped the version of <http://sigs.k8s.io/controller-runtime|sigs.k8s.io/controller-runtime>
which included a breaking change made in https://github.com/kubernetes-sigs/controller-runtime/releases/tag/v0.15.0.
Expected behavior
Secret injection works.
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
11/30/2023, 11:04 PMGitHub
11/30/2023, 11:04 PM<https://github.com/flyteorg/flyte/tree/master|master>
by EngHabu
<https://github.com/flyteorg/flyte/commit/05418513ae140ed69c55842634956c650f4ac6ea|05418513>
- Remove deprecated InjectDecoder (#4507)
flyteorg/flyteGitHub
11/30/2023, 11:11 PM$HOME
references in the config file.
☑︎ Fix the webhook configuration namespace to use POD_NAMESPACE
downstream API
Check all the applicable boxes
☑︎ I updated the documentation accordingly.
☑︎ All new and existing tests passed.
☑︎ All commits are signed-off.
flyteorg/flyte
GitHub Actions: build-and-push-sandbox-bundled-image
Codecov: 59.36% (-0.04%) compared to 0541851
Codecov: 20.00% of diff hit (target 59.40%)
✅ 27 other checks have passed
27/30 successful checksGitHub
11/30/2023, 11:35 PM<https://github.com/flyteorg/flyte/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flyte/commit/64f097fe0e80cdf44250e04ca861038d5012d845|64f097fe>
- Fix /Users/haytham resolution and webhook namespace (#4509)
flyteorg/flyteGitHub
11/30/2023, 11:50 PMGitHub
12/01/2023, 12:15 AM<https://github.com/flyteorg/flyte/tree/master|master>
by jeevb
<https://github.com/flyteorg/flyte/commit/a8f4904353634b34a435402766bdca5d6d7a76b8|a8f49043>
- Add note on updating sandbox cluster configuration (#4510)
flyteorg/flyteGitHub
12/01/2023, 12:34 AM<https://github.com/flyteorg/flytekit/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytekit/commit/a34ab93532f32a5c28ff65f72a7a7212e986e77c|a34ab935>
- Add Why, What, Tests Sections
in PR Template (#2003)
flyteorg/flytekitGitHub
12/01/2023, 1:26 AMGitHub
12/01/2023, 1:29 AMGitHub
12/01/2023, 1:52 PMGitHub
12/01/2023, 1:58 PMGitHub
12/01/2023, 2:27 PMarray node map task
will failed with following Error:
Workflow[flytesnacks:development:my_project.workflows.maptask_with_TFJob_v2.mnist_tensorflow_workflow] failed.
RuntimeExecutionError: max number of system retry attempts [11/10] exhausted.
Last known status message: failed at Node[n0].
RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [tensorflow]: number of worker should be more then 0
It seems that the wrong custom data passing to tensorflow plugin.
Expected behavior
Pass with no error
Additional context to reproduce
1. Activate demo cluster with tensorflow plugin enable.
2. pyflyte run following sample code.
from flytekit import task, workflow, ImageSpec
from flytekit.experimental import map_task
from flytekitplugins.kftensorflow import TfJob, Worker, Chief, PS, Evaluator
from typing import Optional, List
image_spec = ImageSpec(packages=["flytekit", "flytekitplugins-kftensorflow", "tensorflow"], registry="chaoheng"
, registry_config="/home/chaoheng/.docker/config.json")
task_config = TfJob(
worker=Worker(replicas=1),
chief=Chief(replicas=1),
ps=PS(replicas=1),
evaluator=Evaluator(replicas=1),
)
@task(task_config=task_config, container_image=image_spec)
def t3(a: int) -> int:
return a + 1
@workflow
def wf() -> List[Optional[int]]:
mt = map_task(t3)(a=[1, 2, 3])
return mt
Screenshots
image
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
12/01/2023, 2:29 PMget_custom()
with python_function_task
in array_node_map_task.py
How was this patch tested?
Testing kftensorflow with array node in sandbox.
image
Screenshots
image
Check all the applicable boxes
☐ I updated the documentation accordingly.
☑︎ All new and existing tests passed.
☑︎ All commits are signed-off.
Related PRs
Docs link
flyteorg/flytekit
✅ All checks have passed
2/2 successful checksGitHub
12/01/2023, 7:42 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flytekit/commit/5cc1b711889aaf178980ef3bec908f004033cdfa|5cc1b711>
- Override the container image of map task (#2011)
flyteorg/flytekitGitHub
12/01/2023, 7:57 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytekit/commit/4b37b9d17bab09809f4ba0b0310fd50397a24d31|4b37b9d1>
- Introduce class decorator to flyin vscode for url exposure (#1994)
flyteorg/flytekitGitHub
12/01/2023, 9:49 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flytekit/commit/5c6802c97388c40d0431e4fc308e2ed651528952|5c6802c9>
- use original file/directory with local pyflyte run (#1957)
flyteorg/flytekitGitHub
12/01/2023, 10:01 PMGitHub
12/01/2023, 10:12 PMoperationId
it represents.
2. We will fetch all execution spans using depth=0
and will revisit this to better optimize performance in later versions if this feature starts gaining traction.
3. The hover tool-tip will be handled in a separate ticket.
--- The following is out of date in the listed cases ---
The new UI will be controlled by a toggle ("View Details") and will break up the graph into sections. The colors for these new buckets should be opacity values on black/white overlayed over the existing phase color. Crucially, these overlayed values should be consistent across events (eg, the overlay treatment applied to "plugin" durations should be the same regardless of phase)
Image▾
Image▾
Image▾
GitHub
12/01/2023, 10:13 PMGitHub
12/01/2023, 10:38 PMGitHub
12/01/2023, 11:46 PM<https://github.com/flyteorg/flyte/tree/master|master>
by Future-Outlier
<https://github.com/flyteorg/flyte/commit/ebed2a658dc9743d44d1601dcdbbeac762072eba|ebed2a65>
- add template (#4512)
flyteorg/flyteGitHub
12/01/2023, 11:52 PMGitHub
12/02/2023, 3:53 AMGitHub
12/02/2023, 5:00 AMDEFAULT_CODE_SERVER_DIR_NAME
with DEFAULT_CODE_SERVER_DIR_NAMES
• Replace DEFAULT_CODE_SERVER_REMOTE_PATH
with DEFAULT_CODE_SERVER_REMOTE_PATHS
• Modify `flytekit-flyin/flytekitplugins/flyin/vscode_lib/config.py`:
• Replace code_server_dir_name
with code_server_dir_names
• Replace code_server_remote_path
with code_server_remote_paths
• Modify `flytekit-flyin/flytekitplugins/flyin/vscode_lib/constants.py`:
• Replace DEFAULT_CODE_SERVER_REMOTE_PATH
with DEFAULT_CODE_SERVER_REMOTE_PATHS
• Replace DEFAULT_CODE_SERVER_DIR_NAME
with DEFAULT_CODE_SERVER_DIR_NAMES
• Modify `flytekit-flyin/flytekitplugins/flyin/vscode_lib/decorator.py`:
• Add get_code_server_info
function
• Replace config.code_server_remote_path
with code_server_remote_path
• Replace config.code_server_dir_name
with code_server_dir_name
How was this patch tested?
• Modify test_flyin_plugin.py
in the plugins/flytekit-flyin/tests
directory
• Add DEFAULT_CODE_SERVER_DIR_NAMES
and DEFAULT_CODE_SERVER_REMOTE_PATHS
to the flytekit-flyin
plugin configuration
• Add mock_code_server_info_dict
function in the plugins/flytekit-flyin/tests/test_flyin_plugin.py
• Add test_arm_platform
function in the plugins/flytekit-flyin/tests/test_flyin_plugin.py
• Add test_amd_platform
function in the plugins/flytekit-flyin/tests/test_flyin_plugin.py
• Add test_platform_unsupported
function in the plugins/flytekit-flyin/tests/test_flyin_plugin.py
Setup process
Screenshots
Check all the applicable boxes
☐ I updated the documentation accordingly.
☑︎ All new and existing tests passed.
☑︎ All commits are signed-off.
Related PRs
Docs link
flyteorg/flytekit
✅ All checks have passed
2/2 successful checksGitHub
12/02/2023, 7:38 AMflytectl register
supports the usage of macros like {{ registration.project }}
in the python code, whereas pyflyte register
does not. this capability can be particularly helpful when working with reference tasks or launch plans.
Goal: What should the final outcome look like, ideally?
pyflyte register
needs to support macros.
Describe alternatives you've considered
the current workaround is to use flytectl register
.
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyte