GitHub
07/10/2023, 2:57 PM<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>
by cosmicBboy
<https://github.com/flyteorg/flyte-conference-talks/commit/9562ad0e1afb4f3e5fa1354caebfabdb4e8ee0ad|9562ad0e>
- update notebook
flyteorg/flyte-conference-talksGitHub
07/10/2023, 4:02 PMGitHub
07/10/2023, 4:05 PMGitHub
07/10/2023, 5:15 PMGitHub
07/10/2023, 5:15 PMGitHub
07/10/2023, 6:08 PM<https://github.com/flyteorg/flyte/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flyte/commit/e6bd94a4746d3b411fd69e72bef88cdb5bbd1fdf|e6bd94a4>
- Streamline sandbox and functional tests to avoid disk pressure on CI VM (#3828)
flyteorg/flyteGitHub
07/10/2023, 6:17 PM$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: flyte-binary-inline-config-secret
namespace: flyte
type: Opaque
stringData:
202-database-secrets.yaml: |
database:
postgres:
password: <DB_PASSWORD>
203-storage-secrets.yaml: |
storage:
stow:
config:
access_key_id: <S3_ACCESS_KEY>
secret_key: <S3_SECRET_KEY>
204-auth-secrets.yaml: |
auth:
appAuth:
selfAuthServer:
staticClients:
flytepropeller:
client_secret: <CLIENT_SECRET_HASH>
EOF
2. Reference the newly created secret in .Values.configuration.inlineSecretRef
in values.yaml
as follows:
configuration:
inlineSecretRef: flyte-binary-inline-config-secret
The above secrets will be merged into the base configuration created by the chart, and values will be available, as expected, to the Flyte platform.
OIDC/Internal Client secrets
1. Create an external secret containing the client secret values:
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: flyte-binary-client-secrets-external-secret
namespace: flyte
type: Opaque
stringData:
client_secret: <INTERNAL_CLIENT_SECRET>
oidc_client_secret: <OIDC_CLIENT_SECRET>
EOF
2. Reference the newly created secret in .Values.configuration.auth.clientSecretsExternalSecretRef
in values.yaml
as follows:
configuration:
auth:
clientSecretsExternalSecretRef: flyte-binary-client-secrets-external-secret
flyteorg/flyte
GitHub Actions: build-and-push-sandbox-bundled-image
✅ 11 other checks have passed
11/12 successful checksGitHub
07/10/2023, 6:59 PMGitHub
07/10/2023, 7:00 PMGitHub
07/10/2023, 7:18 PMGitHub
07/10/2023, 7:32 PM<https://github.com/flyteorg/flytectl/tree/master|master>
by hamersaw
<https://github.com/flyteorg/flytectl/commit/35904329a4f8b3676c3debad240fba63e2342e82|35904329>
- Adding support for ArrayNode (#406)
flyteorg/flytectlGitHub
07/10/2023, 7:56 PMGitHub
07/10/2023, 7:56 PM<https://github.com/flyteorg/homebrew-tap/tree/main|main>
by flyte-bot
<https://github.com/flyteorg/homebrew-tap/commit/ddddaf5c05140ad40b83490d2b6050867dbad97e|ddddaf5c>
- Brew formula update for flytectl version v0.6.42
flyteorg/homebrew-tapGitHub
07/10/2023, 8:47 PM<https://github.com/flyteorg/flyte/tree/master|master>
by jeevb
<https://github.com/flyteorg/flyte/commit/71028141a7d94761fe7003076abfbc35befb663b|71028141>
- Add support for leveraging pre-created secrets in flyte-binary helm chart (#3807)
flyteorg/flyteGitHub
07/10/2023, 9:15 PM<https://github.com/flyteorg/flyteconsole/tree/master|master>
by ursucarina
<https://github.com/flyteorg/flyteconsole/commit/f6d5fb97ce2b934e92b8ec95041c5f6f3babd9f1|f6d5fb97>
- fix: node executions list going blank (#788)
flyteorg/flyteconsoleGitHub
07/10/2023, 9:30 PMGitHub
07/10/2023, 10:19 PMGitHub
07/10/2023, 11:34 PMflyte-binary
Helm chart and pulling in some comments from https://github.com/flyteorg/flyte/pull/3363/files
flyteorg/flyte
GitHub Actions: build-and-push-sandbox-bundled-image
✅ 11 other checks have passed
11/12 successful checksGitHub
07/11/2023, 12:32 AMGitHub
07/11/2023, 12:59 AMGitHub
07/11/2023, 1:16 AM<https://github.com/flyteorg/flytekit/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytekit/commit/b26cd2a47e32318b85de4c058278234c3ac827b9|b26cd2a4>
- update docs around pyflyte run (#1728)
flyteorg/flytekitGitHub
07/11/2023, 1:19 AMGitHub
07/11/2023, 3:05 AMGitHub
07/11/2023, 4:54 AMGitHub
07/11/2023, 5:33 AM<https://github.com/flyteorg/flytekit/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytekit/commit/982ffe6097dd6700d084ff4da276fe4315927e77|982ffe60>
- fix Image Spec requirements.txt hash (#1731)
flyteorg/flytekitGitHub
07/11/2023, 2:57 PM<https://github.com/flyteorg/flytestdlib/tree/master|master>
by EngHabu
<https://github.com/flyteorg/flytestdlib/commit/907929bcae2a45492ab9b5be0fa96419abe043a2|907929bc>
- Stow head etag (#160)
flyteorg/flytestdlibGitHub
07/11/2023, 3:04 PMGitHub
07/11/2023, 3:56 PMShellTask
. As of now, ShellTask
error message is not helpful since it doesn't capture the stderr
from the subprocess. Additionally, the error thrown by ShellTask
is currently classified as system error instead of user error.
Type
☑︎ Bug Fix
☐ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☑︎ Smoke tested
☑︎ Unit tests added
☑︎ Code documentation added
Complete description
• Raise FlyteRecoverableError
from ShellTask
so that it's classified as user's error and the retry policy specified in the task is respected.
• Improve the error message captured by ShellTask
. As of now the error message generated by ShellTask
in flyteconsole looks as follows. Notice that it doesn't contain any details of the failed command.
[3/3] currentAttempt done. Last Error: SYSTEM::Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/flytekit/core/base_task.py", line 527, in dispatch_execute
raise e
File "/usr/local/lib/python3.8/site-packages/flytekit/core/base_task.py", line 524, in dispatch_execute
native_outputs = self.execute(**native_inputs)
File "/usr/local/lib/python3.8/site-packages/flytekit/extras/tasks/shell.py", line 220, in execute
subprocess.check_call(gen_script, shell=True)
File "/usr/local/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
Message:
Command '
export RUN_DATE=2023-07-04 &&
make run
' returned non-zero exit status 2.
SYSTEM ERROR! Contact platform administrators.
With this change, users would be able to see stdout
and stderr
of the subprocess as part of the error message.
Tracking Issue
flyteorg/flyte#3559
Follow-up issue
N.A.
flyteorg/flytekit
GitHub Actions: build-plugins (3.8, flytekit-onnx-scikitlearn)
GitHub Actions: build-plugins (3.8, flytekit-onnx-pytorch)
GitHub Actions: build-plugins (3.8, flytekit-modin)
GitHub Actions: build-plugins (3.8, flytekit-mlflow)
GitHub Actions: build-plugins (3.8, flytekit-kf-tensorflow)
GitHub Actions: build-plugins (3.8, flytekit-kf-pytorch)
GitHub Actions: build-plugins (3.8, flytekit-kf-mpi)
GitHub Actions: build-plugins (3.8, flytekit-k8s-pod)
GitHub Actions: build-plugins (3.8, flytekit-huggingface)
GitHub Actions: build-plugins (3.8, flytekit-hive)
GitHub Actions: build-plugins (3.8, flytekit-greatexpectations)
GitHub Actions: build-plugins (3.8, flytekit-envd)
GitHub Actions: build-plugins (3.8, flytekit-duckdb)
GitHub Actions: build-plugins (3.8, flytekit-dolt)
GitHub Actions: build-plugins (3.8, flytekit-deck-standard)
GitHub Actions: build-plugins (3.8, flytekit-dbt)
GitHub Actions: build-plugins (3.8, flytekit-data-fsspec)
GitHub Actions: build-plugins (3.8, flytekit-dask)
GitHub Actions: build-plugins (3.8, flytekit-bigquery)
GitHub Actions: build-plugins (3.8, flytekit-aws-sagemaker)
GitHub Actions: build-plugins (3.8, flytekit-aws-batch)
GitHub Actions: build-plugins (3.8, flytekit-aws-athena)
GitHub Actions: lint
GitHub Actions: build (windows-latest, 3.11)
GitHub Actions: build (windows-latest, 3.9)
GitHub Actions: build (windows-latest, 3.8)
GitHub Actions: build (ubuntu-latest, 3.11)
GitHub Actions: build (ubuntu-latest, 3.10)
GitHub Actions: build (ubuntu-latest, 3.9)
GitHub Actions: build (ubuntu-latest, 3.8)
✅ No checks have passed
0/30 successful checksGitHub
07/11/2023, 4:04 PMrecord_outputs
. This is a problem when we try to use the output of a notebook task, e.g. as the return output of the workflow
"""An example of how to use Jupyter Notebooks in your workflows."""
from pathlib import Path
from typing import Tuple
from flytekit import workflow, kwtypes
from flytekit.types.file import FlyteFile
from flytekit.types.structured import StructuredDataset
from flytekitplugins.papermill import NotebookTask
from workflows.example_00_intro import get_data
data_analysis_task = NotebookTask(
name="workflows.data_analysis_task",
notebook_path=str(
Path(__file__).parent.parent.absolute()
/ "notebooks"
/ "data_analysis_task.ipynb"
),
render_deck=True,
disable_deck=False,
inputs=kwtypes(structured_dataset=StructuredDataset),
outputs=kwtypes(success=bool),
)
@workflow
def data_analysis_wf() -> Tuple[bool, FlyteFile, FlyteFile]:
data = get_data()
return data_analysis_task(structured_dataset=data)
Trying to pyflyte register
this workflow will result on the following error:
RPC Failed, with Status: StatusCode.INTERNAL
details: failed to compile workflow for [resource_type:WORKFLOW project:"flytesnacks" domain:"development" name:"workflows.example_notebook_tasks.data_analysis_wf" version:"uMWWUPTSJ7aW7oSK79MRDg==" ] with err failed to compile workflow with err Collected Errors: 4
Error 0: Code: MismatchingTypes, Node Id: end-node, Description: Variable [out_nb] (type [blob:<format:"ipynb" > ]) doesn't match expected type [blob:<> ].
Error 1: Code: MismatchingTypes, Node Id: end-node, Description: Variable [out_rendered_nb] (type [blob:<format:"html" > ]) doesn't match expected type [blob:<> ].
Error 2: Code: ParameterNotBound, Node Id: end-node, Description: Parameter not bound [o1].
Error 3: Code: ParameterNotBound, Node Id: end-node, Description: Parameter not bound [o2].
On the other hand, changing the signature to the one explicit output:
def data_analysis_wf() -> bool:
We get this error:
Failed with Unknown Exception <class 'AssertionError'> Reason: The Workflow specification indicates only one return value, received 3
Expected behavior
Flyte should gracefully handle NotebookTask outputs somehow.
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
07/11/2023, 4:37 PM