flyte-org #flyte-github

GitHub

09/11/2023, 11:48 PM

<https://github.com/flyteorg/flytekit/tree/master|master>

by wild-endeavor

<https://github.com/flyteorg/flytekit/commit/173704811fae7cab64b0eb4a27d68f6948b0e9e4|17370481>

- Feat: Add type support for pydantic BaseModels (#1660) flyteorg/flytekit

GitHub

09/11/2023, 11:48 PM

#1660 Feat: Add type support for pydantic BaseModels Pull request opened by ArthurBook TL;DR Lets you use pydantic basemodels as inputs and outputs in tasks. Also extends the basemodels to have any flyte supported file as field. Type ☐ Bug Fix ☑︎ Feature ☑︎ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☐ Any pending items have an associated Issue Complete description Added serializer/deserializers for flytepath and flytedirectory s.t. pydantic can jsonify these objects too. Type Example

Copy code

from pydantic import BaseModel

class Config(BaseModel):
    lr: float = 1e-3
    batch_size: int = 32
    files: List[FlyteFile]
    directories: List[FlyteDirectory]

@task
def train(cfg: Config):
    ...

Tracking Issue flyteorg/flyte#2686 Follow-up issue NA flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

GitHub

09/12/2023, 1:04 AM

#2652 [Docs] Docs on how Propeller ensures a task is executed exactly once Issue created by sonjaer Description I was curious how Propeller ensures a task is executed exactly once? I was curious since we have non idempotent tasks that we wouldn't want to run twice. Is there any instance when this could not be true? When deploying a new Propeller or error writing to etcd? Is this something that could be added to the Propeller Architecture docs here Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

09/12/2023, 1:04 AM

#2660 [Docs] Document how to log with CloudWatch and Fluentd Issue created by marc-union We need better coverage of connecting Flyte logging with AWS CloudWatch and Fluentd Potential location for this new content is in this page or a sibling page: https://docs.flyte.org/projects/cookbook/en/latest/auto/deployment/configure_logging_links.html Related: Slack conversation: https://flyte-org.slack.com/archives/C01P3B761A6/p1656366983976689 Issue: #2635 https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html https://docs.flyte.org/en/latest/deployment/cluster_config/flytepropeller_config.html flyteorg/flyte

GitHub

09/12/2023, 1:04 AM

#2679 [Docs] Guide on how to write Flytekit shim plugins for Backend plugins Issue created by cosmicBboy Description Currently, we have docs for how to write backend plugins but don't have any guides for how to write a flytekit shim plugin that allows flytekit task authors to use that backend plugin. An example of this is the Sagemaker Training plugin, which requires both. This issue breaks up the tasks described in #1541 Context:

https://www.youtube.com/watch?v=59w9HYOcJJ8&t=1282s▾

Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

09/12/2023, 1:04 AM

#2680 [Docs] Guide on how to write SQL plugins Issue created by cosmicBboy Description We have several SQL-related plugins like the SQL, Snowflake, and BigQuery plugins, but we need an extension guide as to how to write flytekit operators for other SQL-like packages. This issue breaks up the tasks described in #1541 Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

09/12/2023, 1:04 AM

#2681 [Docs] Guide on how to write data persistence plugins Issue created by cosmicBboy Description We currently support S3, GCS, and Local FS data persistence for when Flyte has to serialize data that's passed from one task to another, we should write a guide on how to write additional ones for e.g. Azure's blob store, etc. This issue breaks up the tasks described in #1541 Example: https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-data-fsspec Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

09/12/2023, 1:04 AM

#2694 [Docs] Publish Code Style Guide Issue created by brucearctor Description It is great for a project to have consistent code formatting/style. This is often accomplished by a linting, and as part of pre-commit tests. But, it would also be helpful if a styleguide were published to make the choices/opinions explicit. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

09/12/2023, 1:04 AM

#2696 [Update] M1 doesn't support generating API docs Issue created by SmritiSatyanV Description Users of M1 mac can't execute

make generate

command to generate API docs in repos such as FlytePropeller, Flyteidl and so on. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

09/12/2023, 1:04 AM

#2707 [Core feature] Speed up Deployment of a cluster - esp. for getting started or testing Issue created by brucearctor Motivation: Why do you think this is important? Takes ~30 minutes to go through the various stages. Explore, and ideally find ways, to get that time much less. Goal: What should the final outcome look like, ideally? Faster cluster [ EKS ] and Flyte specific starting times. Describe alternatives you've considered Not doing anything. Propose: Link/Inline OR Additional context Imagining this will either involve finding opta optimizations, or dropping into Terraform to reduce the number of stages, while being explicit about

depends_on

. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

09/12/2023, 1:04 AM

#2710 Entity-relationship diagram of workflows, tasks, launch plans, and executions Issue created by wild-endeavor flyteorg/flyte

GitHub

09/12/2023, 1:04 AM

#2719 [BUG] Demo cluster not starting on slow internet connection Issue created by nanohanno Describe the bug Starting the demo cluster via

flytectl demo start

returned after ~5 min with this error message

Copy code

+---------------------------------------------+---------------+-----------+
|                   SERVICE                   |    STATUS     | NAMESPACE |
+---------------------------------------------+---------------+-----------+
| flyte-kubernetes-dashboard-7fd989b99d-hgmqb | Pending       | flyte     |
+---------------------------------------------+---------------+-----------+
| minio-55b8c8f4bc-mvjz5                      | Pending       | flyte     |
+---------------------------------------------+---------------+-----------+
| postgres-bdb75f779-cngdp                    | Running       | flyte     |
+---------------------------------------------+---------------+-----------+
Error: Get "<https://127.0.0.1:30086/api/v1/nodes>": dial tcp 127.0.0.1:30086: connect: connection refused

Running

flytectl demo exec -- kubectl describe pod -n flyte

shows that the pending pods are pulling the image before it exits. Also, it turned out that my internet connection was slow but increasing

$FLYTE_TIMEOUT

as recommended in #2197 did not help. Could it be that it fails while waiting for the deployments to be ready ( flyte/docker/sandbox-lite/flyte-entrypoint-dind.sh Line 63 in </flyteorg/flyte/commit/cf24edfbb8c55be5d29c96f7f6ba761ceb44003f|cf24edf> ) when it is still loading the image? The timeout

--timeout=5m

is not affected by changing

$FLYTE_TIMEOUT

I guess. Expected behavior The demo cluster starts succesfully. Additional context to reproduce 1. Slow internet connection :) 2.

flytectl demo start

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

09/12/2023, 1:04 AM

#2723 [BUG] Flytectl crashes on registration of workflow definitions coming from flytekit-java Issue closed as not planned by github-actions[bot] flyteorg/flyte

GitHub

09/12/2023, 1:04 AM

#2724 [UI Feature] Console should show resource names in task requests/limits Issue created by katrogan Motivation: Why do you think this is important? It's difficult to debug what resource requests or limits are from the task sidebar when we use the enum name, e.g.

Screen Shot 2022-07-27 at 2 19 54 PM▾

Goal: What should the final outcome look like, ideally? Instead of the enum value we should show the resource name Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

09/12/2023, 1:04 AM

#2727 [BUG] GCP Opta Deployment does not set roles and permissions Issue created by tszumowski Describe the bug When I run the opta GCP deployment steps, everything gets deployed including the service accounts. However, it did not apply the Roles or Permissions described here. I needed to manually follow those directions after running the Opta deployment in order for Flyte to work. Expected behavior Created GCP Service Accounts should have the permissions required to read/write from buckets as well as have the FlyteAdmin role should have the iam.serviceAccounts.signBlob applied. All permissions defined in the manual instructions should be in the deployment. Additional context to reproduce See this thread for details. To reproduce, deploy the Opta GCP configuration and check the service account permissions for FlyteAdmin. It should have signBlob shown above. I attached the logs from my run. opta_gcp_deploy_logs.txt Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

09/12/2023, 1:04 AM

#2798 [Plugin] Weights and Biases Integration Issue created by cosmicBboy Users can already use weights and biases in their Flyte workflows. For example, we have a tutorial that uses W&B for metrics tracking Similar to the mlflow integration proposal, #2797, it would be worth creating an Integration subsection for W&B, with the simplest case as the first example. However, there's also an opportunity here for syntactic sugar to help users initialize, configure, and log metrics and artifacts in a similar way to the MLFLow integration proposal so that users can benefit from e.g. tying W&B runs with Flyte metadata like workflow execution ids, etc. so that we can encode "best practices" of how to use Flyte + W&B in our plugin SDK. API Proposal 1: Decorator Plugin Use the task decorator and/or workflow decorator pattern to create a more seamless experience. This would introduce a new plugin pattern flytekit, which modifies the underlying function wrapped by

@task

and

@workflow

. Example

Copy code

import mlflow
import flytekitplugins.wandb
from flytekit import task, dynamic

@dynamic
@flytekitplugins.wandb.experiment(
    # TBD: figure out what experiment-level configurations can be automatically
    # handled by Flyte, e.g. determining a project name that defaults to "{workflow_name}-{execution_id}"
)
def model_experiment(hyperparameter_grid: List[dict]):
    models = []
    data = ...
    for hyperparameters in hyperparameter_grid:
        models.append(train_model(hyperparameters=hyperparameters, data=data))
    ...

@task
@flytekitplugins.wandb.run(
    # TBD: figure out which wandb.init options would make sense here
    # <https://docs.wandb.ai/ref/python/init>
    # The project name will default to the parent workflow's project name.
)
def train_model(hyperparameters: dict, data: ...):
    # follow the wandb integrations guides based on ML framework of choice:
    # <https://docs.wandb.ai/guides/integrations>
    model = MySklearnModel(**hyperparameters)
    ... # fit
    
    wandb.log({"key": value})

    return model

API Proposal 2: extend
@task
and
@workflow
arguments Task config plugins don't really make sense for MLFlow experiment tracking/logging, since the

task_config

argument is typically used for task types that have specific backend resource requirements (e.g. Spark, Ray, MPI tasks) and is orthogonal to configuring experiments and logging metrics. Therefore, to support similar functionality to proposal 1, we could introduce additional arguments to the

@task

and

@workflow

decorators, e.g.

Copy code

import mlflow
from flytekitplugins.wandb import RunConfig, ExperimentConfig
from flytekit import task, dynamic

@dynamic(..., logging_config=ExperimentConfig(...))
def model_experiment(hyperparameter_grid: List[dict]):
    models = []
    data = ...
    for hyperparameters in hyperparameter_grid:
        models.append(train_model(hyperparameters=hyperparameters, data=data))
    ...

import mlflow
from flytekitplugins.wandb import RunConfig, ExperimentConfig
from flytekit import task, workflow

@task(..., logging_config=RunConfig(...))
def train_model(hyperparameters: dict):
    model = MySklearnModel(**hyperparameters)
    ... # fit
    return model

flyteorg/flyte

GitHub

09/12/2023, 1:04 AM

#2810 [Core feature] Flytekit plugin termination hook Issue created by hajapy Motivation: Why do you think this is important? As a flytekit plugin author, I would like to be able to define cleanup actions (in python) that should be run when a flyte task or workflow is aborted (or terminates unexpectedly). For example, this could be used to let the plugin cancel long-running operations in external services if a user aborts a workflow execution. Goal: What should the final outcome look like, ideally? The flytekit PythonTask interface could expose a new method (analogous to execute) that would be called upon termination of the task via an external signal. Describe alternatives you've considered An effort was made to use the stdlib's signal module to register handlers from the custom plugin module, but this didn't ultimately work (the signal seemed to be intercepted by pyflyte-execute). Propose: Link/Inline OR Additional context This suggestion was discussed in a flyte slack thread. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

09/12/2023, 1:04 AM

#2811 [Plugin] Plotly Flyte deck renderer Issue created by cosmicBboy Flyte supports visualizing arbitrary artifacts in html form via Flyte Decks. Currenty the

flytekitplugins-deck-standard

contains a few renderers for dataframes, markdown, and box plots. The purpose of the issue is to add another renderer class for arbitrary plotly graphs. flyteorg/flyte

GitHub

09/12/2023, 1:04 AM

#2812 [Core feature] Explore FlyteAdmin slow SQL Issue created by katrogan Motivation: Why do you think this is important? A running Flyte deployment logs frequent slow SQL:

Copy code

flyteadmin-794f779986-mqrs7 flyteadmin 
flyteadmin-794f779986-mqrs7 flyteadmin 2022/08/24 19:09:37 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/workflow_repo.go:79 SLOW SQL >= 200ms
flyteadmin-794f779986-mqrs7 flyteadmin [703.604ms] [rows:1] SELECT * FROM "workflows" WHERE project = 'flytesnacks' AND domain = 'development' AND name = 'core.control_flow.high_cpu_wf.single_integer_map_task' ORDER BY created_at desc LIMIT 1
flyteadmin-794f779986-mqrs7 flyteadmin 
flyteadmin-794f779986-mqrs7 flyteadmin 2022/08/24 19:09:37 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/workflow_repo.go:79 SLOW SQL >= 200ms
flyteadmin-794f779986-mqrs7 flyteadmin [720.675ms] [rows:1] SELECT * FROM "workflows" WHERE project = 'flytesnacks' AND domain = 'development' AND name = '<http://core.control_flow.dynamics.wf|core.control_flow.dynamics.wf>' ORDER BY created_at desc LIMIT 1
flyteadmin-794f779986-mqrs7 flyteadmin 
flyteadmin-794f779986-mqrs7 flyteadmin 2022/08/24 19:09:37 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/workflow_repo.go:79 SLOW SQL >= 200ms
flyteadmin-794f779986-mqrs7 flyteadmin [747.358ms] [rows:1] SELECT * FROM "workflows" WHERE project = 'flytesnacks' AND domain = 'development' AND name = 'core.control_flow.conditions.multiplier_2' ORDER BY created_at desc LIMIT 1
flyteadmin-794f779986-mqrs7 flyteadmin 
flyteadmin-794f779986-mqrs7 flyteadmin 2022/08/24 19:09:37 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/launch_plan_repo.go:133 SLOW SQL >= 200ms
flyteadmin-794f779986-mqrs7 flyteadmin [483.240ms] [rows:1] SELECT "launch_plans"."id","launch_plans"."created_at","launch_plans"."updated_at","launch_plans"."deleted_at","launch_plans"."project","launch_plans"."domain","launch_plans"."name","launch_plans"."version","launch_plans"."spec","launch_plans"."workflow_id","launch_plans"."closure","launch_plans"."state","launch_plans"."digest","launch_plans"."schedule_type" FROM "launch_plans" inner join workflows on launch_plans.workflow_id = workflows.id WHERE launch_plans.project = 'flytesnacks' AND launch_plans.domain = 'development' AND launch_plans.name = 'core.control_flow.chain_entities.chain_workflows_wf' LIMIT 1
flyteadmin-794f779986-mqrs7 flyteadmin 
flyteadmin-794f779986-mqrs7 flyteadmin 2022/08/24 19:09:37 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/launch_plan_repo.go:133 SLOW SQL >= 200ms
flyteadmin-794f779986-mqrs7 flyteadmin [485.314ms] [rows:1] SELECT "launch_plans"."id","launch_plans"."created_at","launch_plans"."updated_at","launch_plans"."deleted_at","launch_plans"."project","launch_plans"."domain","launch_plans"."name","launch_plans"."version","launch_plans"."spec","launch_plans"."workflow_id","launch_plans"."closure","launch_plans"."state","launch_plans"."digest","launch_plans"."schedule_type" FROM "launch_plans" inner join workflows on launch_plans.workflow_id = workflows.id WHERE launch_plans.project = 'flytesnacks' AND launch_plans.domain = 'development' AND launch_plans.name = 'core.control_flow.chain_tasks.chain_tasks_wf' LIMIT 1
flyteadmin-794f779986-mqrs7 flyteadmin 
flyteadmin-794f779986-mqrs7 flyteadmin 2022/08/24 19:09:37 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/launch_plan_repo.go:133 SLOW SQL >= 200ms
flyteadmin-794f779986-mqrs7 flyteadmin [406.597ms] [rows:1] SELECT "launch_plans"."id","launch_plans"."created_at","launch_plans"."updated_at","launch_plans"."deleted_at","launch_plans"."project","launch_plans"."domain","launch_plans"."name","launch_plans"."version","launch_plans"."spec","launch_plans"."workflow_id","launch_plans"."closure","launch_plans"."state","launch_plans"."digest","launch_plans"."schedule_type" FROM "launch_plans" inner join workflows on launch_plans.workflow_id = workflows.id WHERE launch_plans.project = 'flytesnacks' AND launch_plans.domain = 'development' AND launch_plans.name = '<http://control_flow.large_fanout_lp.wf|control_flow.large_fanout_lp.wf>' LIMIT 1

we should investigate indices or other performance optimizations to improve the query performance Goal: What should the final outcome look like, ideally? Fast queries to select workflows and launch plans Describe alternatives you've considered slower performance Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

09/12/2023, 1:04 AM

#2814 [BUG] flyteadmin make generate has different results between go 1.18 and 1.19 Issue created by andrewwdye Describe the bug Running

make generate

on a clean flyteadmin repo produces different results between go versions 1.18 and 1.19. This causes the Check Go Generate checker to fail and prevents using the latest version of go for development. Looks like this is related to

enumer

and alvaroloes/enumer#68. Expected behavior Running

make generate

on a clean flyteadmin repo with the latest version of go produces the same output as Check Go Generate. Additional context to reproduce 1. Install latest version of go (1.19) 2. From a clean flyteadmin repo, run

make generate

3. Run

git diff

Copy code

diff --git a/auth/config/authorizationservertype_enumer.go b/auth/config/authorizationservertype_enumer.go
index a5c7dc2..f6e89a6 100644
--- a/auth/config/authorizationservertype_enumer.go
+++ b/auth/config/authorizationservertype_enumer.go
@@ -1,6 +1,5 @@
 // Code generated by "enumer --type=AuthorizationServerType --trimprefix=AuthorizationServerType -json"; DO NOT EDIT.

-//
 package config

 import (
diff --git a/auth/config/samesite_enumer.go b/auth/config/samesite_enumer.go
index af9bfdf..e42e58f 100644
--- a/auth/config/samesite_enumer.go
+++ b/auth/config/samesite_enumer.go
@@ -1,6 +1,5 @@
 // Code generated by "enumer --type=SameSite --trimprefix=SameSite -json"; DO NOT EDIT.

-//
 package config

 import (
diff --git a/pkg/runtime/interfaces/inlineeventdatapolicy_enumer.go b/pkg/runtime/interfaces/inlineeventdatapolicy_enumer.go
index 63ff94e..7c3895b 100644
--- a/pkg/runtime/interfaces/inlineeventdatapolicy_enumer.go
+++ b/pkg/runtime/interfaces/inlineeventdatapolicy_enumer.go
@@ -1,6 +1,5 @@
 // Code generated by "enumer -type=InlineEventDataPolicy -trimprefix=InlineEventDataPolicy"; DO NOT EDIT.

-//
 package interfaces

 import (

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

09/12/2023, 1:04 AM

#2815 [Housekeeping] flyteadmin should validate that goimports was run Issue created by andrewwdye Describe the issue We should add a checker to ensure that

goimports

has run, similar to Check Go Generate, and based on guidance in the README. What if we do not do this? Anyone contributing to Flyte may start from an unclean

goimports

state Related component(s) flyteadmin Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

09/12/2023, 1:04 AM

#2821 [Docs] An example showing AWS EFS volume in eks cluster for ``ReadWriteMany`` Issue created by SmritiSatyanV Description Provide an example that illustrates how EFS volume can be used in eks cluster. Ketan: I think we should have an example of using Efs Reference Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

09/12/2023, 1:04 AM

#2851 Scaling a repo: add pyflyte register details to getting started guide Pull request opened by shivaylamba on <!date^1662478448^{date_short}|2022-09-06T15:34:08Z> Signed-off-by: shivay shivaylamba@gmail.com This resolves issue #2828 Scaling a repo: add pyflyte register to getting started guide flyteorg/flyte GitHub Actions: docs ✅ 4 other checks have passed 4/5 successful checks

GitHub

09/12/2023, 1:04 AM

#2830 add custom dependencies guide Pull request opened by shivaylamba Signed-off-by: shivay shivaylamba@gmail.com This resolves issue #2825 Add a separate file for handling custom dependencies. flyteorg/flyte GitHub Actions: docs ✅ 4 other checks have passed 4/5 successful checks

GitHub

09/12/2023, 1:04 AM

#2832 [Core feature] Specifying TTL for flyte managed files Issue created by hajapy Motivation: Why do you think this is important? Data that is passed from tasks in flyte workflows is typically in the form of a FlyteFile, which abstracts away how that data is stored on a remote object store, like s3. Regulatory requirements can often impose a lifetime on data, and there is no obvious way to specify the lifetime of such files that are managed by Flyte. By supporting a specification like a time to live (TTL) at the Flyte abstraction layer, it would be possible as a user to apply these types of expirations to objects in a granular manner (only expire data that needs to be expired) in a cloud-agnostic way and ultimately enable users to meet their regulatory requirements. Goal: What should the final outcome look like, ideally? Ideally, when constructing a FlyteFile or FlyteDirectory, the user could pass in an optional TTL or expiry timestamp. Once that object gets created by a flyte workflow, the flyte backend would be aware of it and via a periodic job clean up the object(s) once they have expired. A record of the expiration action, obtainable through flyte's apis or visible somewhere in the flyteconsole, would assist in auditability and compliance efforts. Past workflows that are either re-run or otherwise inspected should handle the fact that some object(s) may be expired. Describe alternatives you've considered 1. Apply lifetime policies to objects in the flyte backend blob store. This would delete objects that are not actually needing to be deleted, so not optimal. 2. Apply lifetime policies to objects in the flyte backend object store in conjunction with some tag filtering whereby Flyte would still somehow know to tag certain objects but delegate the cleanup to the backend store. This is not as granular as the proposal and would need cloud-vendor specific implementations. One also needs to coordinate the bucket configuration with the Flyte configuration to get this to work. In both cases, Flyte may be unaware of those objects being deleted by the configured backend object store and lead to a suboptimal user experience interacting with past workflows where those objects were passed / output. Propose: Link/Inline OR Additional context Suggested in slack thread Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

09/12/2023, 1:04 AM

#2850 Add Guide: tasks beyond python Pull request opened by shivaylamba Signed-off-by: shivay shivaylamba@gmail.com This resolves issue #2826 Add a separate file for Spark task flyteorg/flyte GitHub Actions: docs ✅ 4 other checks have passed 4/5 successful checks

GitHub

09/12/2023, 1:05 AM

#2852 [Core feature] Use fast_package in pyflyte run Issue created by pingsutw Motivation: Why do you think this is important? Currently, pyflyte run use register_script, and it has some limitations. such as 1. Failed to run the workflow on the remote cluster if workflow script imports other files because pyflyte run only uploads a single script to blob store. 2. It can't register a LaunchPlan. now, it will only register tasks and workflows. We can add a new flag

--pkg

in the

pyflyte run

, and use

fast_package

to register the entire package (directory) if people use this flag. Goal: What should the final outcome look like, ideally?

pyflyte run --remote --pkg example.py wf

Use

pkg

plag to fast register the entire package. Otherwise, register a single script by default. Describe alternatives you've considered Replace

register_script

with fast_package Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

09/12/2023, 1:05 AM

#2853 [Docs] pyflyte register examples Issue created by SmritiSatyanV Description To improve searchability of

pyflyte register

, add more examples mentioning

pyflyte register

Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

09/12/2023, 1:05 AM

#2859 [Core feature] Per-domain secrets mounting Issue created by slauth-io-tal Motivation: Why do you think this is important? When running a task within a certain domain (production/staging/development), it should be (security-)isolated from other domains. Meaning, secrets should be bound to a single domain and shouldn't be accessible to other domains. An example - while the same task in different domains might require a database secret, the tasks running in the development domain shouldn't have access to the production database. Nor should a task determine what secret to request/use based on its domain - secrets should be treated like an environment configuration. Goal: What should the final outcome look like, ideally? Tasks are able to request a secret, and during runtime the control plane determines which secret to mount according to the task domain. So, a task running in development and requesting the database secret will get a secret representing the development database. Describe alternatives you've considered Currently implementing this mechanism within the code of tasks. Preferably this'll be included in Flyte itself. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

09/12/2023, 1:05 AM

#2863 [Core feature] Use https://github.com/Textualize/rich for pyflyte commands to help build great UX Issue created by kumare3 Motivation: Why do you think this is important? Sometimes errors are not easy to understand, sometimes stack traces are raised. Also tables and other things can be rendered much better with rich Goal: What should the final outcome look like, ideally? Beautiful rendering of the CLI Describe alternatives you've considered NA Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte