#575 Update flyteplugins to v1.0.67 Pull request opened by bstadlbauer TL;DR What it says on the can flyteorg/flytepropeller GitHub Actions: Build & Push Flytepropeller Image GitHub Actions: Goreleaser GitHub Actions: Bump Version ✅ 11 other checks have passed 11/14 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytepropeller/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flytepropeller/commit/05e837b0810b0f09a205efa4d816d10667cdf923|05e837b0>

- Update flyteplugins to v1.0.67 (#575) flyteorg/flytepropeller

#3765 [Core feature] Support configuring pip index url with the image spec feature Issue created by mucahitkantepe Motivation: Why do you think this is important? We have some private common libraries that we want to access in our Flyte workflows using image spec feature. To be able to use imageSpec feature that was highlighted with Flyte 1.6 release, we need image spec feature to support configuring pip index url. envd does support this, and it should be fairly small change. Goal: What should the final outcome look like, ideally?

image_spec=ImageSpec(
    base_image="some-base-image:latest
     packages=["a-private-library==1.6.0"
     pip_index="<https://private-pip-index/simple>"
)

and the generated build.envd file would look like this

# syntax=v1

def build():
...
    config.pip_index(url = "<https://private-pip-index/simple>")
...

Describe alternatives you've considered I think it's pretty straightforward. Propose: Link/Inline OR Additional context envd has built-in support for this, as shown in the main example in their readme https://github.com/tensorchord/envd#create-an-envd-environment Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

Release - v1.1.96 New release published by flyte-bot Changelog • 05e837b Update flyteplugins to v1.0.67 (#575) flyteorg/flytepropeller

#996 Reference new kubeflow training operator instead of deprecated pytorch operator Pull request opened by fg91 The kubeflow pytorch operator has been replaced by the new training operator which bundles the operators for pytorch, tensorflow, ... into a single one. The pytorch flytesnacks example points to the deprecated pytorch operator repository which might confuse users, see e.g. here. This PR adapts the example to reference the new training operator additionally in order to not make users think the underlying operator is not maintained anymore. I still left the link to the original repository in the docs, despite the repo's deprecation, since the example in the new repository doesn't use Mnist but FashionMnist. flyteorg/flytesnacks GitHub Actions: Build & Push to GHCR (cookbook/integrations/flytekit_plugins/mlflow_example) GitHub Actions: Build & Push to GHCR (cookbook/integrations/flytekit_plugins/greatexpectations) GitHub Actions: Build & Push to GHCR (cookbook/integrations/flytekit_plugins/duckdb_examples) GitHub Actions: Build & Push to GHCR (cookbook/integrations/flytekit_plugins/dolt) GitHub Actions: Build & Push to GHCR (cookbook/integrations/flytekit_plugins/dbt_example) GitHub Actions: Build & Push to GHCR (cookbook/integrations/external_services/databricks) GitHub Actions: Build & Push to GHCR (cookbook/integrations/aws/sagemaker_training) GitHub Actions: Build & Push to GHCR (cookbook/integrations/aws/sagemaker_pytorch) GitHub Actions: Build & Push to GHCR (cookbook/integrations/aws/batch) GitHub Actions: Build & Push to GHCR (cookbook/integrations/aws/athena) GitHub Actions: Build & Push to GHCR (cookbook/core) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/ml_training/spark_horovod) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/ml_training/pima_diabetes) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/ml_training/nlp_processing) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/ml_training/mnist_classifier) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/ml_training/house_price_prediction) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/feature_engineering/feast_integration) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/feature_engineering/eda) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/bioinformatics/blast) GitHub Actions: Docs Warnings ✅ 3 other checks have passed 3/23 successful checks

#3766 [UI Feature] Selecting a version from the static graph should pre-select that version when a user is launching a WF Issue created by jsonporter Motivation: Why do you think this is important? Currently, when viewing a version on the version detail page a user can select version (radio buttons) to update the DAG we render. However, selecting that version does not track that intent to other experiences like launching that workflow - a user has suggested that when a specific version is selected on that view, the launch form should keep track of that and pre-select that version when launching. Goal: What should the final outcome look like, ideally? Selecting a version on the version detail page should pre-select that version if the user clicks "Launch" (from that view) but still reset to default otherwise. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3767 [UI Feature] Extra lines in Graph View ? Issue created by tsheiner Motivation: Why do you think this is important? This is reported by Felix Ruess in Flyte Slack. Felix Ruess Does anyone know why Flyte workflows always seem to have unnecessary connections between tasks? Like A -> B -> C, but also A -> C Dan Rammer (hamersaw) can you elaborate? maybe a code example, i'm really not following. this is just the transitive property, if c depends on b and b depends on a then c will always execute after a right? Felix Ruess Yes, it is just a bit annoying to see that extra line in the DAG visualization... Would look much cleaner if the transitive dependency would not be shown Dan Rammer (hamersaw) oh sure, my understanding is that the UI displays the exact task-level dependencies. for example:

def task_a() -> (int, int):
    # ...

@task
def task_b(m: int) -> int:
    # ...

@task
def task_c(n: int, l: int) -> int:
    # ...

@workflow
def wf() -> int:
    x, y = task_a()
    z = task_b(m=x)
    return task_c(n=y, l=z)

would create exactly what you mentioned. is that not the case? or do you want to UI to automatically remove the a->c line because it's redundant? Felix Ruess yeah, would be nice if the UI would not show the redundant lines... not sure if they are needed in the backend... Ketan (kumare3) Why is it redundant Ketan (kumare3) I guess the lines are showing where data feeds from Ketan (kumare3) Cc @Tim Sheiner Dan Rammer (hamersaw) IMO it would be more confusing to remove the line. Felix Ruess simple example where the extra lines are a bit confusing: Tim Sheiner @Felix Ruess have I correctly labeled the 'extra' lines in your example? Felix Ruess yes Goal: What should the final outcome look like, ideally? TBD, this ticket is about investigating if diagram view could use refinement, understanding why Felix thinks lines are extra, but Dan thinks they are helpful. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1684 Update pythonbuild.yml: add hugging face plugin to CI Pull request opened by cosmicBboy TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin ☑︎ CI Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit Codecov: 71.03% (-0.01%) compared to 8437023 ✅ 29 other checks have passed 29/30 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flyte/commit/d95ee84333af84a327760062b602c4cd29b072bd|d95ee843>

- Update Flyte components (#3764) flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by pingsutw

<https://github.com/flyteorg/flytekit/commit/3370a96fe1df3b484b21f17a5cb919600333ac8c|3370a96f>

- feat: Add Auth0/audience support for ClientCredentials flow (#1639) flyteorg/flytekit

#997 Update tasks_and_workflows.md Pull request opened by hebiao064 Add description about: "Workflow Inputs also considered as Promise within Workflows" flyteorg/flytesnacks ✅ All checks have passed 3/3 successful checks

#1685 envd cache locally Pull request opened by cosmicBboy TL;DR Enable local caching of envd build Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit GitHub Actions: lint GitHub Actions: Docs Warnings ✅ 2 other checks have passed 2/4 successful checks

#3768 Replace installation instructions for pytorch operator with training operator Pull request opened by fg91 Describe your changes This PR updates the installation instructions accordingly. For the tensorflow operator this was already documented correctly. I copied the installation instructions from there. The kubeflow pytorch operator has been deprecated in favor of the new training operator. ☑︎ I updated the documentation accordingly. ☐ All new and existing tests passed. ☑︎ All commits are signed-off. flyteorg/flyte ✅ All checks have passed 6/6 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytesnacks/tree/master|master>

by fg91

<https://github.com/flyteorg/flytesnacks/commit/21a4ce1e27d2ea99f16c33db3a9dce924745f014|21a4ce1e>

- Reference new kubeflow training operator instead of deprecated pytorch operator (#996) flyteorg/flytesnacks

Release - flytesnacks v0.3.198 New release published by flyte-bot Changelog • 21a4ce1 Reference new kubeflow training operator instead of deprecated pytorch operator (#996) flyteorg/flytesnacks

#1686 Improve error message for pyflyte-fast-execute Pull request opened by pingsutw TL;DR Remove CalledProocessError, and only show user error Before: After: Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Here is an example to reproduce it.

from flytekit import task, workflow
import torch

@task
def t1():
    print("helloee")


@workflow
def wf():
    t1()


if __name__ == '__main__':
    wf()

Tracking Issue NA Follow-up issue NA flyteorg/flytekit Codecov: 71.03% (-0.01%) compared to 8437023 ✅ 29 other checks have passed 29/30 successful checks

#1687 Local agent Pull request opened by pingsutw TL;DR Run agent code locally. Add

AsyncAgentExecutorMixin

, and any agent task inherited from it can run locally.

import pandas as pd

from flytekit import kwtypes, StructuredDataset, workflow, task
from flytekitplugins.bigquery import BigQueryTask, BigQueryConfig

bigquery_doge_coin = BigQueryTask(
    name=f"bigquery.doge_coin",
    inputs=kwtypes(version=int),
    query_template="SELECT * FROM `bigquery-public-data.crypto_dogecoin.transactions` WHERE version = @version LIMIT 10;",
    output_structured_dataset_type=StructuredDataset,
    task_config=BigQueryConfig(ProjectID="flyte-test-340607")
)


@task
def t1(sd: StructuredDataset):
    print(sd.open(pd.DataFrame).all())


@workflow
def wf():
    sd = bigquery_doge_coin(version=1)
    bigquery_doge_coin(version=1)
    t1(sd=sd)


if __name__ == '__main__':
    wf()

Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue NA Follow-up issue NA flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by samhita-alla

<https://github.com/flyteorg/flyte/commit/fb6b5c9be6cbd6acb2be5ebf8c84ba06e0dc95e0|fb6b5c9b>

- Replace installation instructions for pytorch operator with training operator (#3768) flyteorg/flyte

#3769 [Core feature] Configuration containing secrets should use Opaque Secrets over ConfigMaps Issue created by EternalDeiwos Motivation: Why do you think this is important? The way secrets are handled in deployment cause several headaches: 1. Secrets are supplied in plain text in Helm configuration; this is a problem when deployments and Helm

values.yaml

contents are stored in a git repository and deployed using an external tool. 2. There are no existing configuration to pull configuration and/or secrets from existing configmaps or secrets; this makes it difficult inject secrets and configuration from a central vault (e.g. Hashicorp Vault). 3. Potentially long-lived secrets are being kept in ConfigMaps instead of Opaque Secrets; this is generally not idiomatic even if the security of the two option were roughly equivalent. This is creating a significant barrier to me deploying a test instance in our environment (bare metal K8s). Goal: What should the final outcome look like, ideally? I would like to supply names of existing secrets to the Helm chart and/or Kustomization without directly exposing secrets in configuration files or Helm values. This will allow incremental improvement of my deployment configuration without needing to expose secrets directly as part of the deployment. Describe alternatives you've considered I've been working on a Kustomization with significant patching to move existing configmaps into secrets... however I am considering a completely parallel deployment because writing the patches takes more time than editing the original files. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1620 Feat: Add pydantic plugin with `BaseModel` type transformer Pull request opened by fg91 on <!date^1683290781^{date_short}|2023-05-05T12:46:21Z> TL;DR The plugin adds type support for pydantic

BaseModel

. Type ☐ Bug Fix ☐ Feature ☑︎ Plugin Are all requirements met? ☐ Code completed: The transformer does not yet support all Flyte types in a pydantic model (e.g. data frames) ☑︎ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description The plugin allows passing pydantic models to and between tasks: Type Example

from pydantic import BaseModel
import flytekitplugins.pydantic
class Config(BaseModel):
    lr: float = 1e-3
    batch_size: int = 32
@task
def train(cfg: Config):
    ...

Tracking Issue flyteorg/flyte#2686 Follow-up issue NA flyteorg/flytekit Codecov: 70.10% (-0.01%) compared to baf0b89 ✅ 29 other checks have passed 29/30 successful checks

#359 fix v1 pytorch job plugin with elastic policy Pull request opened by yubofredwang TL;DR There was a bug that introduced in PR #345. The pytorch v2 task template is handled incorrectly. Added Elastic Config parsing for v1. It follows the exact same handling logic as v0 since they are kept as same as v0. Follow-up PR in flytekit: PR #1690 Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue flyteorg/flyteplugins ✅ All checks have passed 7/7 successful checks

#3770 [BUG] Custom environment variable injection does not work for complex workflow setup Issue created by honnix Describe the bug A highly appreciated feature enabling injecting custom environment variable was added to flyte 1.6.0 release, for example the related change in flyteadmin: flyteorg/flyteadmin#556 We tested this feature on one of our workflows, and it partially worked. For a simple container task, this worked well and we could observed the custom environment variables were injected into the container. For a complex task, in our case a remote launch plan (for a workflow with multiple tasks) backed task, the custom environment variables were not injected into containers that were part of the remote launch plan. Expected behavior Custom environment variables should be injected to all containers launched for a workflow execution, regardless how the workflow is composed. Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#230 Upgrade aws-java-sdk-bom Pull request opened by honnix TL;DR Upgrade aws-java-sdk-bom to get latest features and security related fixes. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description To fix at least https://security.snyk.io/vuln/SNYK-JAVA-COMAMAZONAWS-2952700 Tracking Issue NA Follow-up issue NA flyteorg/flytekit-java ✅ All checks have passed 3/3 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by honnix

<https://github.com/flyteorg/flytekit-java/commit/81391ae64c4e7ff079ecdc3983a179c2dde792f7|81391ae6>

- Upgrade aws-java-sdk-bom (#230) flyteorg/flytekit-java

#231 Remove unused maven build property Pull request opened by honnix TL;DR Remove unused maven build property Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Removed this unused old property since jib maven plugin does not require this. Tracking Issue NA Follow-up issue NA flyteorg/flytekit-java ✅ All checks have passed 3/3 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by honnix

<https://github.com/flyteorg/flytekit-java/commit/89322ff969933970d83a7cfa9891e3683bdf9b6f|89322ff9>

- Remove unused maven build property (#231) flyteorg/flytekit-java

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by github-actions[bot]

<https://github.com/flyteorg/flytekit-java/commit/f465f6a6c4755c0624221b08f0837b782b5b1d4d|f465f6a6>

- [maven-release-plugin] prepare release 0.4.19 flyteorg/flytekit-java

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by github-actions[bot]

<https://github.com/flyteorg/flytekit-java/commit/ac28797b0026dabc3e2c81f89df98b4a171a5b62|ac28797b>

- [maven-release-plugin] prepare for next development iteration flyteorg/flytekit-java

#576 Propagating environment variables through launchplans Pull request opened by hamersaw TL;DR Using the environment variables from the

ExecutionConfig

as defaults for all launchplans started from the execution. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☑︎ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue fixes flyteorg/flyte#3770 Follow-up issue NA flyteorg/flytepropeller GitHub Actions: Build & Push Flytepropeller Image GitHub Actions: Goreleaser GitHub Actions: Bump Version ✅ 11 other checks have passed 11/14 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytepropeller/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flytepropeller/commit/eadf07ba3ac2e54ee4aaf95a98c9941a5a005646|eadf07ba>

- Correctly validating error code on aborting terminal launchplans (#574) flyteorg/flytepropeller