#1354 [BUG] Flyteadmin shouldn't allow launching an execution in archived state Issue created by pmahindrakar-oss Describe the bug Each launch plan can be in one of two states. • Archived • Active Admin exposes an api to toggle between these two states. But in case a user launches an execution for a launchplan then this state flag is ignored. Expected behavior Flyteadmin create execution should throw an error in case the launch plan state is archived. [Optional] Additional context To Reproduce Steps to reproduce the behavior: 1. update the launch plan state to archived 2. create an execution for the same launchplan Screenshots If applicable, add screenshots to help explain your problem. flyteorg/flyte

#1364 [flytekit] pytest-flyte and integration test framework Issue created by wild-endeavor Motivation: Why do you think this is important? This issue should probably be a design doc instead of an issue, but let's start here. We currently have a pytest plugin that is lightly used to do some integration testing in flytekit. There are many places where we could use something like this • Users that want more confidence beyond unit tests that things will work when deployed. • Integration testing flytekit changes using flytesnacks examples. • Integration testing where multiple components are changing. To do this... flyteorg/flyte

#1369 [Core Feature] Capture and save documentation from flytekit entities Issue created by katrogan Motivation: Why do you think this is important? See #531 for a richer discussion on this feature. For this specific issue, we will want to capture user documentation, links to source code and commit version and pipe that through the Flyte system and make entity descriptions for tasks, workflows, launch plans and executions retrievable via a separate endpoint. Goal: What should the final outcome look like, ideally? It should be possible to add RST-formatted documentation inline with flytekit entity definitions or when launching executions from any means and have that information made retrievable by end users. Describe alternatives you've considered Again, see #531 [Optional] Propose: Link/Inline OR Additional context Design doc: https://docs.google.com/document/d/1wvCvM6pLQ17Zr0DcpS9es2vOFQsNeOaNN99ufGw8O1U/edit?usp=sharing flyteorg/flyte

#1388 [Housekeeping] Remove execution time task resource resolution from flyteadmin Issue created by katrogan Describe the Issue As part of #1361 we introduce task resource resolution in propeller, rather than flyteadmin. However to support backwards compatibility during the upgrade process, the existing code in flyteadmin needs to be cleaned up in a subsequent release. What if we do not do this? Convoluted task resource resolution at execution time within flyteadmin that is discarded. Related component flyteadmin flyteorg/flyte

#2563 [BUG] flytectl delete help doesn't match the actual functionality Issue created by honnix Describe the bug

delete Terminates/deletes various Flyte resources such as tasks, workflows, launch plans, executions, and projects.

But

Usage:
  flytectl delete [command]

Available Commands:
  cluster-resource-attribute Deletes matchable resources of cluster attributes.
  execution                  Terminates/deletes execution resources.
  execution-cluster-label    Deletes matchable resources of execution cluster label.
  execution-queue-attribute  Deletes matchable resources of execution queue attributes.
  plugin-override            Deletes matchable resources of plugin overrides.
  task-resource-attribute    Deletes matchable resources of task attributes.
  workflow-execution-config  Deletes matchable resources of workflow execution config.

For example projects cannot be deleted but archived/activated. It was mentioned in #1619 as doc bug but not fixed yet. Expected behavior A more accurate

delete

command help message. Additional context to reproduce 1. flytectl 2. flytectl delete --help Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1818 Add support pandas 2 in flytekit Pull request opened by pingsutw TL;DR Remove pandas from default dependencies, and support pandas 2 in flytekit Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue flyteorg/flyte#3928 Follow-up issue NA flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

#1403 [Housekeeping] Make FlyteAdmin matchable task resources use a structured quantity Issue created by katrogan Describe the Issue Currently the protobuf definition uses a string to represent resource quantities:https://github.com/flyteorg/flyteidl/blob/081a8526ef0f5d4bc8f0db9eade686744ad855b1/protos/flyteidl/admin/matchable_resource.proto#L35 Using a structured quantity message would improve runtime execution by bumping validation to serialization time. What if we do not do this? Short description of potential consequences Related component flyteadmin, flyteidl flyteorg/flyte

#1417 Creating new spark_sql_task Issue created by yunhao-qing Why would this plugin be helpful to the Flyte community _spark_sql_task_ will allow users to submit a query that will run a Spark. It will not necessarily run on native Spark kubernetes operator. It will be similar structure to current _hive_task_ or _presto_task_. Type of Plugin ☑︎ Python/Java interface only plugin ☐ Web Service (e.g. AWS Sagemaker, GCP DataFlow, Qubole etc...) ☐ Kubernetes Operator (e.g. TfOperator, SparkOperator, FlinkK8sOperator, etc...) ☐ Customized Plugin using native kubernetes constructs ☐ Other Can you help us with the implementation? ☑︎ Yes ☐ No flyteorg/flyte

#1420 [Core Feature] Implement recovery for flytekit remote Issue created by katrogan Motivation: Why do you think this is important? Now that recovery mode is implemented, see #1151 it would be helpful to interact with executions by recovering them in flytekit remote. Note, this issue only covers flytekit remote. Goal: What should the final outcome look like, ideally? It should be possible to recover executions using flytekit remote Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered. [Optional] Propose: Link/Inline OR Additional context If you have ideas about the implementation please propose the change. If inline keep it short, if larger then you link to an external document. flyteorg/flyte

#1422 [Core Feature] Support pause and unpause in scheduler in case admin is down Issue created by pmahindrakar-oss Motivation: Why do you think this is important? Scheduler uses admin for sending workflow executions at the scheduled time. But in case of admin being down currently we have a retry mechanism which through configured number of steps using exponential backoff. Admin being down is a special case where all scheduled workflows will go in this retry loop. The proposal is to introduce a pause mechanism for the scheduler where all the existing schedules are paused and resumed when admin is healthy. We will use admins health api to determine this. When unpaused the scheduler will catch up on all the schedules till the pause time cc : @EngHabu @Ketan (kumare3) @evalsocket Goal: What should the final outcome look like, ideally? The scheduler pauses on admin down and unpauses when admin comes back up healthy and catches up on all the schedules. Describe alternatives you've considered The existing retry mechanism works but it unnecessarily causes all go routines looping in retries [Optional] Propose: Link/Inline OR Additional context If you have ideas about the implementation please propose the change. If inline keep it short, if larger then you link to an external document. flyteorg/flyte

#1456 [Core Feature]Generic Schemas should detect the schema at runtime Issue closed as not planned by github-actions[bot] flyteorg/flyte

#1488 Release process summary Pull request opened by EngHabu on <!date^1631820051^{date_short}|2021-09-16T19:20:51Z> flyteorg/flyte ✅ All checks have passed 1/1 successful checks

#1457 [Docs] Improve debugging documentation for Flytepropeller logs Issue created by kumare3 Describe the documentation update FlytePropeller has logs of the following form

{
  "json": {
    "exec_id": "grs2u5m2ns",
    "node": "n0",
    "ns": "flytesnacks-development",
    "res_ver": "137261245",
    "routine": "worker-2",
    "src": "handler.go:313",
    "tasktype": "python-task",
    "wf": "flytesnacks:development:<http://core.control_flow.dynamics.wf|core.control_flow.dynamics.wf>"
  },
  "level": "warning",
  "msg": "No plugin found for Handler-type [python-task], defaulting to [container]",
  "ts": "2021-09-10T15:53:28Z"
}

There should be a document that explains how to filter logs to get only the logs for one specific workflow / execution id etc flyteorg/flyte

#1467 [flytekit] List plugins automatically in pythonbuild.yml Issue created by wild-endeavor Leftover todo from flyteorg/flytekit#644 We should detect when a new plugin is added automatically. flyteorg/flyte

#1496 [Core Feature] Temp Disk Volumes Issue created by vsbus Motivation: Why do you think this is important? We can specify mem/cpu request/limit for the tasks. It will also be good to have a similar feature for disk. Note, k8s v1.21 has beta for Generic Ephemeral Volumes and they seem to provide what is needed. Goal: What should the final outcome look like, ideally? Feature available for all flyte task types. A usage of this feature could look like this

@flytekit.task(
    retries=1,
    requests=flytekit.Resources(cpu="12", mem="32Gi"),
    limits=flytekit.Resources(mem="64Gi", cpu="12"),
    temp_disk_volumes=[flytekit.Volume(size="100G", mount_path="/tmpdisk_01")],
)
def foo():
  download_large_file_to("/tmpdisk_01/abc")

Describe alternatives you've considered Maybe flyte users can do it today with Sidecar task. But having a support of this parameter in every task would bring more convenience. [Optional] Propose: Link/Inline OR Additional context If you have ideas about the implementation please propose the change. If inline keep it short, if larger then you link to an external document. flyteorg/flyte

#1509 [Core Feature][flytekit] Ability to track resource utilization and gpu utilization of a python task as a plugin Issue created by kumare3 Motivation: Why do you think this is important? When users are running tasks that use GPU's, CPUs and Memory, it might be useful to allow some meta-outputs that render this information Goal: What should the final outcome look like, ideally? There are many libraries like https://github.com/gpuopenanalytics/pynvml etc that can be used to collect stats for various resources and we could either stream or outputs these stats as a rendered PNG at the end of the run. The rendering can be done either in python or on the server and the outputs should be meta outputs. Describe alternatives you've considered Use other external libraries like Wandb etc [Optional] Propose: Link/Inline OR Additional context Write a new plugin

flytekit-resource-monitoring

that can load components that can output certain meta outputs. For this, we will need support for meta-outputs. flyteorg/flyte

#1512 [Docs] docs for IAM, service accounts, labels and annotations Issue created by kumare3 Describe the documentation update There is no good reference explaining how to set service accounts, roles, labels, and annotations. GitHub repo(s) flytesnacks flyteorg/flyte

#1520 Add options to configure SendGrid API Key Issue created by ariefrahmansyah Motivation: Why do you think this is important? While working on notifications using GCP and SendGrid (flyteorg/flyteadmin#259), I need to supply the SendGrid API Key. To achieve that, I configure SENDGRID_API_KEY via environment variables. I deployed Flyte using Helm, but since there's no way to provide environment variables via Helm, I need to manually add SENDGRID_API_KEY. Goal: What should the final outcome look like, ideally? To be able to properly configure SendGrid as the email sender. Describe alternatives you've considered Enable user to specify additional environment variables for flyteadmin deployment. flyteorg/flyte

#1523 [Flytectl Feature] flytectl status should show env vars to be exported Issue created by kumare3 Describe the feature/command for flytectl if you use flytectl sandbox, as a user you may lose the information about how to use kubectl & flytectl to connect to flyte-sandbox. We should allows users to recover this using

flytectl sandbox status

Provide a possible output or ux example

$ flytectl sandbox status
...
To Connect to FlyteSandbox Kube cluster export:
...
To connect to FlyteSandbox using flytectl export:
...

flyteorg/flyte

#1539 Add support for Multiple Architecture types Issue created by akhurana001 Motivation: Why do you think this is important? Multiple cloud providers (AWS etc.) have extensive support for ARM based instances now and we want to explore running Flyte workloads on ARM instances due to multiple benefits over x86. In general, the goal is to enable platform admins to configure multiple architecture types as well as Flyte users to select between them based on which one they want to use/image support etc. Goal: What should the final outcome look like, ideally? Platform admins be able to configure multiple architecture support within the same flyte installation and users being able to specify which one to execute on. [Optional] Propose: Link/Inline OR Additional context The goal is to enable Platform admins to maintain both ARM and x86 ASGs on each Kubernetes cluster and to intelligently route workloads to the right ASG based on which architecture the user image was built/tested/registered for. This can be done with adding taints to the ASGs and tolerations/node-selector to pods something Flyte already does for Interruptible enabled Tasks. However, for this use-case, it might be easier for users to tag the whole repo/sha with ARM rather than tagging individual tasks/nodes. I was thinking we can re-use labels support at Launch-Plan level to implement this: Users can set architecture label either at project level via flyte-config or directly at a launch-plan level which will then be routed correctly to the Flyte backend for those workflow versions/sha and corresponding tolerations and node-selector be added to those pods by FlytePropeller/plugins (similar to how Interruptible works). Let me know what your thoughts are on this. flyteorg/flyte

#1546 [Flytectl Feature] Gracefully handle config files with non-standard extensions Issue created by AdrianoKF Describe the feature/command for flytectl We are trying to use

flytectl

inside a Gitlab CI pipeline. Gitlab supports supplying CI variables mounted as files, where the environment variable points to a file containing the value of the CI var - so naturally we tried setting

FLYTECTL_CONFIG

through this mechanism. However, this fails with the following error message when trying to execute any flytectl command:

Error:
Unsupported Config Type ""
ERRO[0000]
Unsupported Config Type ""

Apparently, this error message originates in the spf13/viper library used by flytectl through flytestdlib. The error is raised because viper tries to detect the type of the config file from its extension, which is unset in our CI environment (Gitlab names the file identical to the variable name, e.g. the

FLYTECTL_CONFIG

variable will be stored under

/builds/resources/repo_name.tmp/FLYTECTL_CONFIG

). The error is not specific to Gitlab CI, but can be reproduced by pointing

FLYTECTL_CONFIG

at any file without an extension, see below. Steps to reproduce

# Note that ./config is a valid YAML flytectl config file
$ export FLYTECTL_CONFIG=$PWD/config
$ cat $FLYTYCTL_CONFIG
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///localhost:30081
  insecure: true
logger:
  show-source: true
  level: 0
storage:
  connection:
    access-key: minio
    auth-type: accesskey
    disable-ssl: true
    endpoint: <http://10.32.16.105:30084>
    region: us-east-1
    secret-key: miniostorage
  type: minio
  container: "my-s3-bucket"
  enable-multicontainer: true

$ flytectl get projects
Error:
Unsupported Config Type ""
ERRO[0000]
Unsupported Config Type ""

$ flytectl version
 A new release of flytectl is available: 0.3.4 → v0.3.4

{
  "App": "flytectl",
  "Build": "8ba75a6",
  "Version": "0.3.4",
  "BuildTime": "2021-10-01 15:37:24.613781861 +0200 CEST m=+0.029412676"
}

Proposed fix Viper offers the

SetConfigName

and

SetConfigType

methods which could be used in flytestdlib instead of

SetConfigFile

if the config file is always assumed to be a YAML file. flyteorg/flyte

#1547 [Flytectl Bug] Update check should not reports out-of-date for latest version Issue created by AdrianoKF Describe the bug

flytectl version

reports that a new release is available, even directly after

flytectl upgrade

$ flytectl upgrade 
You have already latest version of flytectl

$ flytectl version

 A new release of flytectl is available: 0.3.4 → v0.3.4

{
  "App": "flytectl",
  "Build": "8ba75a6",
  "Version": "0.3.4",
  "BuildTime": "2021-10-01 16:08:44.983660372 +0200 CEST m=+0.019362263"
}

Minor nitpick: the

flytectl upgrade

error message contains a typo (missing "the") It seems that the version number specified in the build (used as

stdlibversion.Version

in <cmd/version/version.go|`cmd/version/version.go`> incorrectly omits the

v

of the Git tag for the release. This leads to

IsVersionGreaterThan

in <pkg/util/util.go|`pkg/util/util.go`> incorrectly reporting that the latest Github version is greater than the local build version. I haven't fully grasped the release process, but it seems that the Github release action sets the version number without the

v

prefix in the Goreleaser action, which would explain the discrepancy. Expected behavior No newer release should be reported after upgrading to the latest version using

flytectl upgrade

. [Optional] Additional context To Reproduce 1. Upgrade to latest release using

flytectl upgrade

2. Perform update check through

flytectl version

flyteorg/flyte

#1550 [BUG] Unhelpful error message when registering without having run serialize first Issue created by katrogan Describe the bug If I use flytectl to run register without having first run serialize I get this very confusing message

Error: please check your Storage Config. It failed while uploading the source code. initContainer is required even with `enable-multicontainer`

This in no way indicates that the source code artifacts are missing from the serialization stage. Expected behavior Clear error message when there are no serialized pb files to register [Optional] Additional context To Reproduce Steps to reproduce the behavior: 1. 2. Screenshots If applicable, add screenshots to help explain your problem. flyteorg/flyte

#1556 [Plugin]BlazingSQL and Rapids dataframe support Issue created by kumare3 Why would this plugin be helpful to the Flyte community It would be great to have a simple plugin that would allow users to simply write BlazingSQL queries in their workflows to manipulate data-frames on a GPU as part of their Flyte workflows. An example of writing a query from the User point of view could be

task = BlazingSQL(
     input_schema=FlyteSchema,
    outputs=...,
    query=""" SELECT count(*) FROM taxi GROUP BY year(key)""",
    use_gpus=False, # True by default
)

Type of Plugin ☑︎ Python/Java interface only plugin ☐ Web Service (e.g. AWS Sagemaker, GCP DataFlow, Qubole etc...) ☐ Kubernetes Operator (e.g. TfOperator, SparkOperator, FlinkK8sOperator, etc...) ☐ Customized Plugin using native kubernetes constructs ☐ Other flyteorg/flyte

#1561 [Housekeeping] Test Flytectl &amp; Flytekit on Windows Issue created by samhita-alla Describe the Issue We haven't yet thoroughly tested Flytekit and Flytectl on a windows OS. Would love someone with a windows OS to test this out. What if we do not do this? Flyte on windows may not be foolproof. Related component flytekit, flytectl flyteorg/flyte

#1564 When running in a k8s cluster with mutating webhooks, tasks don't finish since the side car is still working [Core Feature] Issue created by NotMatthewGriffin Motivation: Why do you think this is important? Currently when running a flyte task in a kubernetes cluster that uses mutating webhooks to add sidecars to all pods, the task and associated pod will run until terminated in the flyte web console. Inside of the pod, the container actually running the flyte task has completed and exited with a good status but the sidecar container is still running. To work around this a user must specify the task in their workflow as a pod task like

@task(task_config=Pod(pod_spec=pod_spec, pimary_container_name="container_name"))

. This means that users need to know the configuration of the cluster ( like will any sidecars be added automatically) in order to successfully create/run a flyte workflow. Goal: What should the final outcome look like, ideally? Ideally a user of flyte wouldn't need to declare their task as a pod task in order to run it in a kubernetes cluster that uses mutating webhooks to add sidecars to pods. The user could declare their task like

@task

and it would run the flyte task pod until the flyte task container finishes then the rest of the workflow would be run. Describe alternatives you've considered The only other alternative I've considered is continuing to use the workaround of declaring all workflows/tasks as pod tasks. flyteorg/flyte

#1585 [Docs] Document cluster resource controller and its configuration Issue created by kumare3 Description ☐ How to configure cluster resource controller ☐ How to disable it ☐ How to alter the frequency ☐ High level architecture with diagram ☑︎ Are you sure this issue hasn't been raised already? ☑︎ Have you read the Code of Conduct? flyteorg/flyte

#1587 [Docs] Document architecture and design of Flyte co-pilot Issue created by kumare3 Description Flyte co-pilot powers Raw containers and should be documented ☑︎ Are you sure this issue hasn't been raised already? ☑︎ Have you read the Code of Conduct? flyteorg/flyte

#1592 [Core feature] Simple templating for default annotations/labels/env vars Issue created by frsann Motivation: Why do you think this is important? At the moment it's possible to add hard-coded annotations/labels/env vars to all pods or on a task level, but there is no middle ground. There is no easy way configure the default annotations to have project or domain specific keys/values. One way of achieving this is by allowing simple templating in the default annotations/labels/env vars. (Another way would be to allow defining project/domain specific annotations, but that might serve a different use case) Goal: What should the final outcome look like, ideally? In the propeller config, you'd be able to define something like:

default-annotations:
      - annotationKey1: annotationValue1_{PROJECT}_{DOMAIN}   # -> annotationValue1_flytesnacks_development
      - key_{project}: True   # -> key_flytesnacks: True

The idea would be to use predefined variable names (

PROJECT

,

DOMAIN

, `WORKFLOW`(?) ) that are anyway readily available when creating task pods. Describe alternatives you've considered The alternative would be to allow defining project/domain specific keys/values, which would allow more fine-grain control (maybe all projects don't need the key/value defined?) but at the same time would be less flexible with regards to including new projects. It basically serves a slightly different use case. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1594 [Core feature] FlyteIDL should generate protobuf files for R Issue created by kumare3 Motivation: Why do you think this is important? RProtobuf is a package that allows marshalling and unmarshalling protobuf structs to R objects. Doing so would make it possible to write a task natively in R Goal: What should the final outcome look like, ideally? community should be able to implement an R sdk for writing tasks if not complete workflows Describe alternatives you've considered ContainerNative tasks Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte