1 new commit pushed to

<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>

by cosmicBboy

<https://github.com/flyteorg/flyte-conference-talks/commit/9562ad0e1afb4f3e5fa1354caebfabdb4e8ee0ad|9562ad0e>

- update notebook flyteorg/flyte-conference-talks

Pending review on flyteorg/flytekit-python-template - Manage reminder [#40] ImageSpec overhaul (zeryx) a month stale · a month old · Waiting on _ @Niels Bantilan_

#3828 Streamline sandbox and functional tests to avoid disk pressure on CI VM Pull request opened by jeevb flyteorg/flyte GitHub Actions: test-bootstrap GitHub Actions: trigger-sandbox-build GitHub Actions: compile ✅ 2 other checks have passed 2/5 successful checks

#3856 [Core feature] Storage layer doesn't return ETag on HEAD requests Issue created by EngHabu Motivation: Why do you think this is important? ETAG is a way for clients to verify the server received the right bits on Put requests as well as it allows clients to check the existence of files (beyond just the key-string verification). Goal: What should the final outcome look like, ideally? HEAD requests should return the ETAG of a file on successful responses. Describe alternatives you've considered Do a GET then compute the md5 locally. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#160 Stow head etag Pull request opened by EngHabu TL;DR Support returning ETag for in Metadata responses. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☑︎ Any pending items have an associated Issue Tracking Issue fixes flyteorg/flyte#3856 flyteorg/flytestdlib ✅ All checks have passed 5/5 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flyte/commit/e6bd94a4746d3b411fd69e72bef88cdb5bbd1fdf|e6bd94a4>

- Streamline sandbox and functional tests to avoid disk pressure on CI VM (#3828) flyteorg/flyte

#3807 Add support for leveraging pre-created secrets in flyte-binary helm chart Pull request opened by jeevb Adds support for referencing pre-created Secret objects for sensitive configuration variables (e.g. DB password, auth client secrets, etc.). Also moves sensitive configuration into a Secret object as opposed to ConfigMap. Addresses: #3769 Secrets in Flyte configuration 1. Create an external secret containing info such as DB password, S3 access/secret key, client secret hash, etc as follows:

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: flyte-binary-inline-config-secret
  namespace: flyte
type: Opaque
stringData:
  202-database-secrets.yaml: |
    database:
      postgres:
        password: <DB_PASSWORD>
  203-storage-secrets.yaml: |
    storage:
      stow:
        config:
          access_key_id: <S3_ACCESS_KEY>
          secret_key: <S3_SECRET_KEY>
  204-auth-secrets.yaml: |
    auth:
      appAuth:
        selfAuthServer:
          staticClients:
            flytepropeller:
              client_secret: <CLIENT_SECRET_HASH>
EOF

2. Reference the newly created secret in

.Values.configuration.inlineSecretRef

in

values.yaml

as follows:

configuration:
  inlineSecretRef: flyte-binary-inline-config-secret

The above secrets will be merged into the base configuration created by the chart, and values will be available, as expected, to the Flyte platform. OIDC/Internal Client secrets 1. Create an external secret containing the client secret values:

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: flyte-binary-client-secrets-external-secret
  namespace: flyte
type: Opaque
stringData:
  client_secret: <INTERNAL_CLIENT_SECRET>
  oidc_client_secret: <OIDC_CLIENT_SECRET>
EOF

2. Reference the newly created secret in

.Values.configuration.auth.clientSecretsExternalSecretRef

in

values.yaml

as follows:

configuration:
  auth:
    clientSecretsExternalSecretRef: flyte-binary-client-secrets-external-secret

flyteorg/flyte GitHub Actions: build-and-push-sandbox-bundled-image ✅ 11 other checks have passed 11/12 successful checks

#788 fix: node executions list going blank Pull request opened by ursucarina TL;DR Fixes: node execution list disapprearing Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Null ref causes the execution list to be wiped out Tracking Issue NA Follow-up issue NA flyteorg/flyteconsole GitHub Actions: Build Flyteconsole Image / Push to Github Registry GitHub Actions: lint_project GitHub Actions: unit_tests_with_coverage GitHub Actions: extract_branch ✅ 2 other checks have passed 2/6 successful checks

#587 Remove content md5 requirement Pull request opened by wild-endeavor TL;DR Remove the requirement that content md5 is provided. Depends on flyteorg/flytestdlib#160 Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue This is related to flyteorg/flytekit#1674 flyteorg/flyteadmin GitHub Actions: Build & Push Flyteadmin Image GitHub Actions: Build & Push Flytescheduler Image GitHub Actions: Goreleaser GitHub Actions: Bump Version ✅ 8 other checks have passed 8/12 successful checks

#3857 [BUG] Pyflyte Register's zip system is inconsistent Issue created by zeryx Describe the bug Whenever using pyflyte register to zip and register a workflow + tasks to a cluster, there are some gotchas that can lead to unexpected behavior. If you have files that are ignored by a .gitignore at any point within your project - those files will not end up getting zipped up and available within the task container. Expected behavior Either a notification that some files were ignored during pyflyte register (aka; let users know that pyflyte register respects .gitignore / .dockerignore), and/or provide users with a mechanism to override .gitignores. Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytectl/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flytectl/commit/35904329a4f8b3676c3debad240fba63e2342e82|35904329>

- Adding support for ArrayNode (#406) flyteorg/flytectl

Release - v0.6.42 New release published by flyte-bot Changelog • 3590432 Adding support for ArrayNode (#406) flyteorg/flytectl

1 new commit pushed to

<https://github.com/flyteorg/homebrew-tap/tree/main|main>

by flyte-bot

<https://github.com/flyteorg/homebrew-tap/commit/ddddaf5c05140ad40b83490d2b6050867dbad97e|ddddaf5c>

- Brew formula update for flytectl version v0.6.42 flyteorg/homebrew-tap

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by jeevb

<https://github.com/flyteorg/flyte/commit/71028141a7d94761fe7003076abfbc35befb663b|71028141>

- Add support for leveraging pre-created secrets in flyte-binary helm chart (#3807) flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by ursucarina

<https://github.com/flyteorg/flyteconsole/commit/f6d5fb97ce2b934e92b8ec95041c5f6f3babd9f1|f6d5fb97>

- fix: node executions list going blank (#788) flyteorg/flyteconsole

Release - v1.8.6 New release published by flyte-bot 1.8.6 (2023-07-10) Bug Fixes • node executions list going blank (#788) (f6d5fb9) flyteorg/flyteconsole

#1729 Backport 1.6.1 and 1.6.2 to 1.2 Pull request opened by eapolinario TL;DR This PR backports releases 1.6.1 and 1.6.2 to 1.2 release branch Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit GitHub Actions: build-plugins (3.8, flytekit-onnx-scikitlearn) GitHub Actions: build-plugins (3.8, flytekit-onnx-pytorch) GitHub Actions: build-plugins (3.8, flytekit-modin) GitHub Actions: build-plugins (3.8, flytekit-kf-tensorflow) GitHub Actions: build-plugins (3.8, flytekit-kf-pytorch) GitHub Actions: build-plugins (3.8, flytekit-kf-mpi) GitHub Actions: build-plugins (3.8, flytekit-k8s-pod) GitHub Actions: build-plugins (3.8, flytekit-hive) GitHub Actions: build-plugins (3.8, flytekit-greatexpectations) GitHub Actions: build-plugins (3.8, flytekit-envd) GitHub Actions: build-plugins (3.8, flytekit-duckdb) GitHub Actions: build-plugins (3.8, flytekit-dolt) GitHub Actions: build-plugins (3.8, flytekit-deck-standard) GitHub Actions: build-plugins (3.8, flytekit-dbt) GitHub Actions: build-plugins (3.8, flytekit-data-fsspec) GitHub Actions: build-plugins (3.8, flytekit-bigquery) GitHub Actions: build-plugins (3.8, flytekit-aws-sagemaker) GitHub Actions: build-plugins (3.8, flytekit-aws-batch) GitHub Actions: build-plugins (3.8, flytekit-aws-athena) GitHub Actions: build (windows-latest, 3.9) GitHub Actions: build (windows-latest, 3.8) GitHub Actions: build (windows-latest, 3.7) GitHub Actions: build (ubuntu-latest, 3.10) GitHub Actions: build (ubuntu-latest, 3.9) GitHub Actions: build (ubuntu-latest, 3.8) GitHub Actions: build (ubuntu-latest, 3.7) GitHub Actions: lint ✅ 2 other checks have passed 2/29 successful checks

#3858 [Docs] flyte-binary readme and some discussion around secrets Pull request opened by wild-endeavor Docs only change. Adding a readme to the

flyte-binary

Helm chart and pulling in some comments from https://github.com/flyteorg/flyte/pull/3363/files flyteorg/flyte GitHub Actions: build-and-push-sandbox-bundled-image ✅ 11 other checks have passed 11/12 successful checks

#1730 Fix Image spec Hash Pull request opened by RichhLi TL;DR Converting the hash object to string will include the memory address (which differs in every execution), so two identical requirements.txt file will have different hashes. Instead, use hexdigest() for unique hash value. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue flyteorg/flytekit ✅ All checks have passed 2/2 successful checks

#1731 fix Image Spec requirements.txt hash Pull request opened by RichhLi TL;DR Converting the hash object to string will include the memory address (which differs in every execution), so two identical requirements.txt file will have different hashes. Instead, use hexdigest() for unique hash value. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue flyteorg/flytekit GitHub Actions: build-plugins (3.11, flytekit-sqlalchemy) GitHub Actions: build-plugins (3.11, flytekit-snowflake) GitHub Actions: build-plugins (3.11, flytekit-polars) GitHub Actions: build-plugins (3.11, flytekit-papermill) GitHub Actions: build-plugins (3.11, flytekit-pandera) GitHub Actions: build-plugins (3.11, flytekit-kf-tensorflow) GitHub Actions: build-plugins (3.11, flytekit-kf-pytorch) GitHub Actions: build-plugins (3.11, flytekit-kf-mpi) GitHub Actions: build-plugins (3.11, flytekit-k8s-pod) GitHub Actions: build-plugins (3.11, flytekit-huggingface) GitHub Actions: build-plugins (3.11, flytekit-hive) GitHub Actions: build-plugins (3.11, flytekit-envd) GitHub Actions: build-plugins (3.11, flytekit-duckdb) GitHub Actions: build-plugins (3.11, flytekit-deck-standard) GitHub Actions: build-plugins (3.11, flytekit-aws-batch) GitHub Actions: build-plugins (3.10, flytekit-vaex) ✅ 14 other checks have passed 14/30 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by pingsutw

<https://github.com/flyteorg/flytekit/commit/b26cd2a47e32318b85de4c058278234c3ac827b9|b26cd2a4>

- update docs around pyflyte run (#1728) flyteorg/flytekit

#1021 Updated the getting started guide Pull request opened by litmus-zhang I improved the flow of the getting started guide to ensure new users and contributors can set up flyte to follow along with the first example flyteorg/flytesnacks DCO: DCO ✅ 1 other check has passed 1/2 successful checks

#1022 Add a comma for syntax error in launch_plan Pull request opened by Future-Outlier We should add a comman after "schedule=CronSchedule(schedule="*/1 * * * *")" flyteorg/flytesnacks ✅ All checks have passed 2/2 successful checks

#1023 Add a comma for syntax error in launch_plan Pull request opened by Future-Outlier We should add a comman after "schedule=CronSchedule(schedule="*/1 * * * *")" flyteorg/flytesnacks ✅ All checks have passed 3/3 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by pingsutw

<https://github.com/flyteorg/flytekit/commit/982ffe6097dd6700d084ff4da276fe4315927e77|982ffe60>

- fix Image Spec requirements.txt hash (#1731) flyteorg/flytekit

1 new commit pushed to

<https://github.com/flyteorg/flytestdlib/tree/master|master>

by EngHabu

<https://github.com/flyteorg/flytestdlib/commit/907929bcae2a45492ab9b5be0fa96419abe043a2|907929bc>

- Stow head etag (#160) flyteorg/flytestdlib

Release - v1.0.20 New release published by github-actions[bot] Changelog • 907929b Stow head etag (#160) flyteorg/flytestdlib

#1732 Improve error handling in ShellTask Pull request opened by pradithya TL;DR Improve the error returned by

ShellTask

. As of now,

ShellTask

error message is not helpful since it doesn't capture the

stderr

from the subprocess. Additionally, the error thrown by

ShellTask

is currently classified as system error instead of user error. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added Complete description • Raise

FlyteRecoverableError

from

ShellTask

so that it's classified as user's error and the retry policy specified in the task is respected. • Improve the error message captured by

ShellTask

. As of now the error message generated by

ShellTask

in flyteconsole looks as follows. Notice that it doesn't contain any details of the failed command.

[3/3] currentAttempt done. Last Error: SYSTEM::Traceback (most recent call last):

      File "/usr/local/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
        return wrapped(*args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/flytekit/core/base_task.py", line 527, in dispatch_execute
        raise e
      File "/usr/local/lib/python3.8/site-packages/flytekit/core/base_task.py", line 524, in dispatch_execute
        native_outputs = self.execute(**native_inputs)
      File "/usr/local/lib/python3.8/site-packages/flytekit/extras/tasks/shell.py", line 220, in execute
        subprocess.check_call(gen_script, shell=True)
      File "/usr/local/lib/python3.8/subprocess.py", line 364, in check_call
        raise CalledProcessError(retcode, cmd)

Message:

    Command '
        export RUN_DATE=2023-07-04 &&
        make run
    ' returned non-zero exit status 2.

SYSTEM ERROR! Contact platform administrators.

With this change, users would be able to see

stdout

and

stderr

of the subprocess as part of the error message. Tracking Issue flyteorg/flyte#3559 Follow-up issue N.A. flyteorg/flytekit GitHub Actions: build-plugins (3.8, flytekit-onnx-scikitlearn) GitHub Actions: build-plugins (3.8, flytekit-onnx-pytorch) GitHub Actions: build-plugins (3.8, flytekit-modin) GitHub Actions: build-plugins (3.8, flytekit-mlflow) GitHub Actions: build-plugins (3.8, flytekit-kf-tensorflow) GitHub Actions: build-plugins (3.8, flytekit-kf-pytorch) GitHub Actions: build-plugins (3.8, flytekit-kf-mpi) GitHub Actions: build-plugins (3.8, flytekit-k8s-pod) GitHub Actions: build-plugins (3.8, flytekit-huggingface) GitHub Actions: build-plugins (3.8, flytekit-hive) GitHub Actions: build-plugins (3.8, flytekit-greatexpectations) GitHub Actions: build-plugins (3.8, flytekit-envd) GitHub Actions: build-plugins (3.8, flytekit-duckdb) GitHub Actions: build-plugins (3.8, flytekit-dolt) GitHub Actions: build-plugins (3.8, flytekit-deck-standard) GitHub Actions: build-plugins (3.8, flytekit-dbt) GitHub Actions: build-plugins (3.8, flytekit-data-fsspec) GitHub Actions: build-plugins (3.8, flytekit-dask) GitHub Actions: build-plugins (3.8, flytekit-bigquery) GitHub Actions: build-plugins (3.8, flytekit-aws-sagemaker) GitHub Actions: build-plugins (3.8, flytekit-aws-batch) GitHub Actions: build-plugins (3.8, flytekit-aws-athena) GitHub Actions: lint GitHub Actions: build (windows-latest, 3.11) GitHub Actions: build (windows-latest, 3.9) GitHub Actions: build (windows-latest, 3.8) GitHub Actions: build (ubuntu-latest, 3.11) GitHub Actions: build (ubuntu-latest, 3.10) GitHub Actions: build (ubuntu-latest, 3.9) GitHub Actions: build (ubuntu-latest, 3.8) ✅ No checks have passed 0/30 successful checks

#3859 [BUG] NotebookTask output types cannot actually be used Issue created by cosmicBboy Describe the bug `NotebookTask`s have 2 implicit outputs, on top of whatever the user outputs via

record_outputs

. This is a problem when we try to use the output of a notebook task, e.g. as the return output of the workflow

"""An example of how to use Jupyter Notebooks in your workflows."""

from pathlib import Path
from typing import Tuple

from flytekit import workflow, kwtypes
from flytekit.types.file import FlyteFile
from flytekit.types.structured import StructuredDataset
from flytekitplugins.papermill import NotebookTask


from workflows.example_00_intro import get_data


data_analysis_task = NotebookTask(
    name="workflows.data_analysis_task",
    notebook_path=str(
        Path(__file__).parent.parent.absolute()
        / "notebooks"
        / "data_analysis_task.ipynb"
    ),
    render_deck=True,
    disable_deck=False,
    inputs=kwtypes(structured_dataset=StructuredDataset),
    outputs=kwtypes(success=bool),
)

@workflow
def data_analysis_wf() -> Tuple[bool, FlyteFile, FlyteFile]:
    data = get_data()
    return data_analysis_task(structured_dataset=data)

Trying to

pyflyte register

this workflow will result on the following error:

RPC Failed, with Status: StatusCode.INTERNAL
        details: failed to compile workflow for [resource_type:WORKFLOW project:"flytesnacks" domain:"development" name:"workflows.example_notebook_tasks.data_analysis_wf" version:"uMWWUPTSJ7aW7oSK79MRDg==" ] with err failed to compile workflow with err Collected Errors: 4
        Error 0: Code: MismatchingTypes, Node Id: end-node, Description: Variable [out_nb] (type [blob:<format:"ipynb" > ]) doesn't match expected type [blob:<> ].
        Error 1: Code: MismatchingTypes, Node Id: end-node, Description: Variable [out_rendered_nb] (type [blob:<format:"html" > ]) doesn't match expected type [blob:<> ].
        Error 2: Code: ParameterNotBound, Node Id: end-node, Description: Parameter not bound [o1].
        Error 3: Code: ParameterNotBound, Node Id: end-node, Description: Parameter not bound [o2].

On the other hand, changing the signature to the one explicit output:

def data_analysis_wf() -> bool:

We get this error:

Failed with Unknown Exception <class 'AssertionError'> Reason: The Workflow specification indicates only one return value, received 3

Expected behavior Flyte should gracefully handle NotebookTask outputs somehow. Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#6 Update material Pull request opened by cosmicBboy flyteorg/flyte-conference-talks GitHub Actions: build (scipy-2023, 3.10, macos-latest) GitHub Actions: build (scipy-2023, 3.10, ubuntu-latest) GitHub Actions: build (scipy-2023, 3.9, windows-latest) GitHub Actions: build (scipy-2023, 3.9, macos-latest) GitHub Actions: build (scipy-2023, 3.9, ubuntu-latest) GitHub Actions: build (scipy-2023, 3.8, windows-latest) GitHub Actions: build (scipy-2023, 3.8, macos-latest) GitHub Actions: build (scipy-2023, 3.8, ubuntu-latest) ✅ 2 other checks have passed 2/10 successful checks