#3080 [Housekeeping] Issues with bundled demo image Issue created by wild-endeavor on <!date^1668623697^{date_short}|2022-11-16T18:34:57Z> New Bundled Sandbox Image This ticket is to collect issues with the new bundled sandbox environment image. This image is the result of the following PRs: #3066 flyteorg/flytectl#370 #2994 #3137 Cluster Resource Cron • Need more investigation but either it's not running, or not picking up additional configuration. • Manually added more templates to the configmap but none of them were created.

$ kf get cm cluster-resource-templates -o yaml
apiVersion: v1
data:
  ac_spark_role.yaml: |
    apiVersion: <http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>
    kind: Role
    metadata:
      name: spark-role
      namespace: {{ namespace }}
    rules:
    - apiGroups: ["*"]
      resources: ["pods"]
      verbs: ["*"]
    - apiGroups: ["*"]
      resources: ["services"]
      verbs: ["*"]
    - apiGroups: ["*"]
      resources: ["configmaps", "persistentvolumeclaims"]
      verbs: ["*"]
  ad_spark_service_account.yaml: |
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: spark
      namespace: {{ namespace }}
  ae_spark_role_binding.yaml: |
    apiVersion: <http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>
    kind: RoleBinding
    metadata:
      name: spark-role-binding
      namespace: {{ namespace }}
    roleRef:
      apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
      kind: Role
      name: spark-role
    subjects:
    - kind: ServiceAccount
      name: spark
      namespace: {{ namespace }}
  namespace.yaml: |
    apiVersion: v1
    kind: Namespace
    metadata:
      name: {{ namespace }}
kind: ConfigMap

Nits • Logging level should be increased. Misc Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#616 Overwriting of cached task outputs for a single execution #minor Pull request opened by MorpheusXAUT on <!date^1664984334^{date_short}|2022-10-05T15:38:54Z> TL;DR Added checkbox to toggle cache overwriting for a single execution. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☑︎ Any pending items have an associated Issue Complete description flyteconsole now supports overwriting of cached outputs for a single workflow or task execution. A checkbox was added to the launch/relaunch forms, similar to the interruptible override. The protobuf timestamp helper changes were added since my local version of latest flyteadmin and flyteconsole seemed to return `number`s instead of `Long`s, causing the UI to error while converting protobuf timestamps. Thus, a simple type check was added - I can revert this if it's not desired/something broken on my local side. As this uses a currently unmerged `flyteidl` version, the PR is created as a draft (for review) until a new version of

flyteidl

is available. Tracking Issue flyteorg/flyte#2867 Follow-up issue NA flyteorg/flyteconsole GitHub Actions: Push to Github Registry GitHub Actions: Get Release Tag GitHub Actions: Generate Release ✅ 8 other checks have passed 8/11 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by jsonporter

<https://github.com/flyteorg/flyteconsole/commit/88bcfba9a96750884b524670652477c18f96c94d|88bcfba9>

- Overwriting of cached task outputs for a single execution #minor (#616) flyteorg/flyteconsole

#2952 [Core feature] CI/CD for Flyte binary image Issue created by wild-endeavor on <!date^1664915854^{date_short}|2022-10-04T20:37:34Z> Flyte Binary CI/CD Background The Flyte binary combines the Flyte components together into one executable. Something is already updating the relevant go-based components (like how Admin gets bumped here in the Flyte release). But there is no CI/CD to build the image. Let's get this added. Request We'd like to: • Make sure a docker image is built for the Flyte binary and pushed properly with every merge to master/release of the Flyte repo. • This is the dockerfile that's being used to create the image today. • The Docker image relies on the latest Flyte console but it should really pick up a constant version of console, specified... somehow. • The

flyte

binary should probably be built and released somewhere as well. Would be nice to be able to

brew install

it. Related to #2993 flyteorg/flyte

#3048 [UI Feature] Distinguish between task and workflow executions in the UI Issue created by katrogan Motivation: Why do you think this is important? In the list executions view it would be useful to differentiate whether an execution belongs to a single task or workflow. Goal: What should the final outcome look like, ideally? Clear to pinpoint the original type of entity used to launch an execution. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#660 Distinguish between task and workflow executions in the UI Issue created by katrogan From flyteorg/flyte#3048 flyteorg/flyteconsole

1 new commit pushed to

<https://github.com/flyteorg/flytesnacks/tree/master|master>

by cosmicBboy

<https://github.com/flyteorg/flytesnacks/commit/173aaa2c5986d2b015329720ab3cbc984b98369a|173aaa2c>

- Update lp_schedules.py (#936) flyteorg/flytesnacks

#3213 Add conditional around the admin secrets Pull request opened by wild-endeavor Tracking issue #3141 Describe your changes Check all the applicable boxes ☐ I updated the documentation accordingly. ☐ All new and existing tests passed. ☐ All commits are signed-off. flyteorg/flyte

Release - flytesnacks v0.3.159 New release published by flyte-bot Changelog • 173aaa2 Update lp_schedules.py (#936) flyteorg/flytesnacks

#507 wip: Add user name to UserInfoResponse header Pull request opened by katrogan Signed-off-by: Katrina Rogan katroganGH@gmail.com TL;DR Builds on #488 to also forward along the fullname of the user. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue NA Follow-up issue NA flyteorg/flyteadmin ✅ All checks have passed 1/1 successful checks

#1406 Read structured dataset from a folder Pull request opened by pingsutw TL;DR

Hugginface

and

polars

transformer failed to read parquet file generated from other transformers, because those transformers always read the data from a specific path (.../00000), but we may have different file name in that folder. We should read the data by default from a folder. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue flyteorg/flyte#3189 Follow-up issue NA flyteorg/flytekit

#3214 [Docs] Create integration docs page for MLFlow integration Issue created by cosmicBboy Description The purpose of this task is to document the MLFlow integration flyteorg/flytekit#1274 in the flytesnacks integration page: https://docs.flyte.org/projects/cookbook/en/latest/integrations.html This should include: • how to install and enable the plugin • how to use the mlflow autolog decorator • visuals on how to view the logs on Flyte Deck • high-level instructions on how to view the logs on an MLflow UI Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by cosmicBboy

<https://github.com/flyteorg/flytekit/commit/55b6602a7d4e3491d5175c6d9039369c2880fc98|55b6602a>

- Add support MLFlow plugin (#1274) flyteorg/flytekit

#1274 Add support MLFlow plugin Pull request opened by kumare3 on <!date^1667345012^{date_short}|2022-11-01T23:23:32Z> Signed-off-by: Ketan Umare ketan.umare@gmail.com TL;DR MLFlow plugin for flytekit. Render mlflow metrics and parameters on the flyte decks. Example:

from flytekit import task, workflow
from flytekitplugins.mlflow import mlflow_autolog
import mlflow
import tensorflow as tf


@task(disable_deck=False)
@mlflow_autolog(framework=mlflow.keras)
def train_model(epochs: int):
    # Refer to <https://www.tensorflow.org/tutorials/keras/classification>
    fashion_mnist = tf.keras.datasets.fashion_mnist
    (train_images, train_labels), (_, _) = fashion_mnist.load_data()
    train_images = train_images / 255.0

    model = tf.keras.Sequential([
        tf.keras.layers.Flatten(input_shape=(28, 28)),
        tf.keras.layers.Dense(128, activation='relu'),
        tf.keras.layers.Dense(10)
    ])

    model.compile(optimizer='adam',
                  loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
                  metrics=['accuracy'])
    model.fit(train_images, train_labels, epochs=epochs)


@workflow
def ml_pipeline(epochs: int):
    train_model(epochs=epochs)


if __name__ == "__main__":
    ml_pipeline()

Type ☐ Bug Fix ☑︎ Feature ☑︎ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Tracking Issue https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

#1407 pr into #1384 Pull request opened by wild-endeavor Please see the other PR flyteorg/flytekit

1 new commit pushed to

<https://github.com/flyteorg/flytepropeller/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flytepropeller/commit/d2098fc5d3a7fa7a94a3332560b27c36f0456126|d2098fc5>

- Add dask plugin (#508) flyteorg/flytepropeller

#3215 [BUG] flyte workflow execution nodes for dynamic tasks collapse in web UI after completion Issue created by mbuchove Dynamic workflows (using the

@dynamic

decorator) will expand out the dynamically created tasks in the web UI after they are run, and will be displayed even after completion: however, after that same execution is refreshed, that expansion into nodes collapses, and the information of the workflow structure is no longer available: here is example code that defines the overall dynamic workflow:

@dynamic
def train_workflow_inner(
    recipe_path: str,
    k_folds: int = 1,
) -> int:

    data_flytedir = data_prep_task(recipe_path=recipe_path, recipe_hash=recipe_hash)

    s = None
    for i in range(k_folds):
        model_and_data_flytedir = training_task(
                data=data_flytedir,
                k_folds=k_folds,
                i_fold=i,
        )
        s = eval_task(data=model_and_data_flytedir)

    return s

flyteorg/flyte

Release - v1.1.59 New release published by flyte-bot Changelog • d2098fc Add dask plugin (#508) flyteorg/flytepropeller

#3216 Build flyte-binary image during sandbox-bundled build Pull request opened by jeevb on <!date^1673068514^{date_short}|2023-01-07T05:15:14Z> flyteorg/flyte GitHub Actions: trigger-sandbox-lite-build GitHub Actions: Functional test GitHub Actions: trigger-single-binary-build ✅ 7 other checks have passed 7/10 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by jeevb

<https://github.com/flyteorg/flyte/commit/0d9aa541afbf6c63a56e1a922bf2e14c078f8637|0d9aa541>

- Build flyte-binary image during sandbox-bundled build (#3216) flyteorg/flyte

#662 test publish Pull request opened by jeevb on <!date^1673052893^{date_short}|2023-01-07T00:54:53Z> flyteorg/flyteconsole DCO: DCO ✅ 2 other checks have passed 2/3 successful checks

#3217 Test functional test Pull request opened by jeevb on <!date^1673070644^{date_short}|2023-01-07T05:50:44Z> flyteorg/flyte GitHub Actions: one GitHub Actions: one GitHub Actions: one GitHub Actions: one GitHub Actions: one GitHub Actions: one GitHub Actions: one GitHub Actions: one GitHub Actions: one GitHub Actions: one GitHub Actions: trigger-sandbox-lite-build ✅ 18 other checks have passed 18/29 successful checks

#508 Publish multi-architecture image Pull request opened by jeevb on <!date^1673073142^{date_short}|2023-01-07T06:32:22Z> Depends on: flyteorg/flytetools#70 flyteorg/flyteadmin GitHub Actions: Build & Push Flyteadmin Image GitHub Actions: Goreleaser GitHub Actions: Build & Push Flytescheduler Image GitHub Actions: Bump Version GitHub Actions: End2End Test GitHub Actions: Integration Test GitHub Actions: Docker Build Images / Build Docker Image ✅ 7 other checks have passed 7/14 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by cosmicBboy

<https://github.com/flyteorg/flytekit/commit/fcf6dcefe36220c244b03f76c16dfdd772d6379b|fcf6dcef>

- fix remote API reference (#1405) flyteorg/flytekit

1 new commit pushed to

<https://github.com/flyteorg/blog-posts-backup/tree/main|main>

by samhita-alla

<https://github.com/flyteorg/blog-posts-backup/commit/7c468a12a6d617221c965ffbc10dfcc1bf57493e|7c468a12>

- create post: The Three Virtues of MLOps: Velocity, Validation and Versioning flyteorg/blog-posts-backup

#3090 [Housekeeping] flyte.org, image incubation project =&gt; gratuated project Issue created by andife on <!date^1668888880^{date_short}|2022-11-19T20:14:40Z> Describe the issue Flyte seems to be a "gratuated project" at lfaidata. The landing page flyte.org still has an image with incubation project. What if we do not do this? webpage might be a bit longer outdated Related component(s) No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by github-actions[bot]

<https://github.com/flyteorg/flytekit-java/commit/c6bbe6ed0bd051028c24d5e7a8f724adab6197f1|c6bbe6ed>

- [maven-release-plugin] prepare release 0.3.28 flyteorg/flytekit-java

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by github-actions[bot]

<https://github.com/flyteorg/flytekit-java/commit/674f07999b6c9a6f46dce535b299b500457eab50|674f0799>

- [maven-release-plugin] prepare for next development iteration flyteorg/flytekit-java

#163 Update maven central URL Pull request opened by honnix TL;DR Update maven central URL. This one refreshes faster. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Tracking Issue NA Follow-up issue NA flyteorg/flytekit-java

Release - 0.3.28 New release published by honnix • Auto generate node ID #162 flyteorg/flytekit-java