#1848 [Core feature] [flytekit] FlyteRemote workflow execution ux flow Issue created by wild-endeavor Motivation: Why do you think this is important? Improve UX for FlyteRemote workflow executions. Currently after calling

remote.fetch_workflow_execution()

you get back a

FlyteWorkflowExecution

instance. The data model backing this object mirrors the responses returned by Admin which was a good first start but we should improve the ux flow for humans. Goal: What should the final outcome look like, ideally? Rather than

we = remote.fetch_workflow_execution()
we.node_executions

the user should be able to do

we.nodes  # to get a map of the nodes in the execution
we.nodes["n1"]  # to get a node
we.nodes["n1"].execution.tasks
we.nodes["n1"].execution.subs??

For a more complicated example, let's say we have

@workflow
def my_wf(val: int) -> int:
    result = square(val=val)
    return result

@dynamic
def run_subwfs() -> typing.Tuple[int, int]:
    x = my_wf(val=5)
    y = my_wf(val=7)
    return x, y


@workflow
def run_subwfs_wf() -> typing.Tuple[int, int]:
    return run_subwfs()

If we ran

run_subwfs_wf

and fetched the execution for it, we'd be able to navigate it like so:

we = r.fetch_workflow_execution()

we.nodes["n0"].kind <- pattern matches

# normal task
we.nodes["n0"].task.executions

# static subwf
we.nodes["n0"].workflow
we.nodes["n0"].workflow.nodes

# static lp
we.nodes["n0"].child_execution

# dynamic
we.nodes["n0"].dynamic.task.executions
we.nodes["n0"].dynamic.workflow.nodes

# branch
we.nodes["n0"].branch.conditions[0].node # really the node execution, .state
we.nodes["n0"].branch.conditions[0].expression  # result or expression

consider FlyteNode -> FlyteNodeExecution Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1832 [Docs] Set of Examples to test launching in UI Issue created by kumare3 Description To Launch an execution, the Console first loads the LaunchPlan and then Creates an Execution using the ExecutionSpec. There should be examples that excercise various scenarios to ease the development and testing. Examples cases Launch plans that are already defined with: (we should consider negative cases as well, where some of them are not defined) 1. Metadata (notiifcations, labels) 2. fixed inputs 3. default inputs 4. ... The UI should allow adding all ExecutionSpec parameters. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1820 [BUG] Flytectl does not show correct state of a scheduled launch plan Issue created by pradithya Describe the bug

flytectl get launchplan

command does not show the correct state of an activated launch plan. Expected behavior The activated launch plan state should be correctly returned by flytectl Additional context to reproduce flytectl version v0.3.20 Screenshots Screen.Recording.2021-11-11.at.10.11.40.PM.mov Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1791 [Core feature] Migrate ArrayJob to use TaskTemplate Config and deprecate ArrayJob proto Issue created by EngHabu Motivation: Why do you think this is important? ArrayJob proto has 4 different primitive values that can be filled in. We can cleanup the dependency on a proto and move these params to configs on the TaskTemplate. Goal: What should the final outcome look like, ideally? Deprecate ArrayJob proto and leverage TaskTemplate.Config map instead. This frees up the Custom field to the individual tasks' which opens up the possibility of defining Array Jobs on custom task types. Describe alternatives you've considered N/A Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1781 [Docs] pyflyte command not found Issue created by ekhc I've been carefully following the Getting Started guide but after brew installing flytectl pyflyte is not found as a command and I don't understand how to access the pyflyte cli bundled with Flytekit flyteorg/flyte

#1760 [BUG] FlytePropeller failed to get permissions to S3 on EKS, error is silent in the background Issue created by kumare3 Describe the bug In case where flytepropeller on EKS, is unable to get permissions to write to S3, it keeps on trying, upto system retries, without visibility in the UI. The workflow remains in

UNKNOWN

status. Ideal, would be if the workflow gets information about the system error being experienced.

{"json":{"exec_id":"uaben0erib","ns":"flytesnacks-development","res_ver":"52746460","routine":"worker-8","src":"executor.go:355","wf":"flytesnacks:development:core.control_flow.high_cpu_wf.single_integer_map_task"},"level":"info","msg":"Handling Workflow [uaben0erib], id: [project:\"flytesnacks\" domain:\"development\" name:\"uaben0erib\" ], p [Ready]","ts":"2021-10-29T04:17:44Z"}
{"json":{"exec_id":"uaben0erib","ns":"flytesnacks-development","res_ver":"52746460","routine":"worker-8","src":"executor.go:122","wf":"flytesnacks:development:core.control_flow.high_cpu_wf.single_integer_map_task"},"level":"info","msg":"Setting the MetadataDir for StartNode [s3://..../metadata/propeller/flytesnacks-development-uaben0erib/start-node/data]","ts":"2021-10-29T04:17:44Z"}
{"json":{"exec_id":"uaben0erib","node":"start-node","ns":"flytesnacks-development","res_ver":"52746460","routine":"worker-8","src":"protobuf_store.go:77","wf":"flytesnacks:development:core.control_flow.high_cpu_wf.single_integer_map_task"},"level":"error","msg":"Failed to write to the raw store [s3://.../metadata/propeller/flytesnacks-development-uaben0erib/start-node/data/0/outputs.pb] Error: Failed to write data [28b] to path [metadata/propeller/flytesnacks-development-uaben0erib/start-node/data/0/outputs.pb].: PutObject, putting object: WebIdentityErr: failed fetching WebIdentity token: \ncaused by: WebIdentityErr: unable to read file at /var/run/secrets/eks.amazonaws.com/serviceaccount/token\ncaused by: open /var/run/secrets/eks.amazonaws.com/serviceaccount/token: permission denied","ts":"2021-10-29T04:17:44Z"}

Expected behavior The UI should show the error that is experienced by the system Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1751 [Housekeeping] enhance grafana metrics explanation Issue created by ncouronne Describe the issue Grafana dashboard for flyte are great but some metrics are hard to understand: For example: What is the unit ops/s ? How to interpret the value 0,0040 ops/s which in this case correspond to one workflow execution ? What if we do not do this? Monitor flyte and get metrics on the executed workflows is crucial for production exploitation of flyte, but more crucial is to get realistic and understandable metrics The result may be that the metrics cannot be used and dev user will not be able to monitor easily the platform. In consequence it could lead dev user to create their proper metrics or consider flyte as a hard platform to monitor without personal dev. Related component(s) grafana dashboard from flyteadmin and flytepropreller exposed metrics Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1731 [Docs] Install instructions for flytekit-java Issue created by EngHabu Description I can't seem to find a doc that details how to get started with flytekit-java (how to install it,... etc.) Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1709 [Housekeeping] make helm should update boiler plate and also bring latest version of helm-docs Issue created by pmahindrakar-oss Describe the issue CI runs

DELTA_CHECK=true make helm

to find any diff in the generated helm and doc files or any other uncommitted generated files. When the same is done from local machine its possible the tools needed to run these are older and don't give the same results as in the CI. While working on GCP deployment for flyte found the generated files had a diff in CI whereas locally it would work fine. The discrepancy was due to older version of helm-docs which was used for generating docs. make update_boilerplate was expected to solve this problem but it didn't What if we do not do this? Developer has to figure out the difference in failure in CI and local environments which can be cumbersome if the tool versions used in the two environments are different. Related component(s) flyte Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1699 [BUG] Deprecated path to helm chart in opta/flyte.yaml Issue created by RustamovAzer Describe the bug Paths in

opta/flyte.yaml

lead to nonexistent folder

helm

flyte/opta/flyte.yaml Line 46 in </flyteorg/flyte/commit/a8cb42015adda0e901d263a67a26def4d704cea2|a8cb420> flyte/opta/flyte.yaml Line 50 in </flyteorg/flyte/commit/a8cb42015adda0e901d263a67a26def4d704cea2|a8cb420> Expected behavior

"../helm"

in both paths should be changed either to

"../charts/flyte"

or

"../charts/flyte-core"

(not sure which one) Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1607 [Housekeeping] Flytekit: Move versions into __init__ files of the root package as well as plugins Issue created by EngHabu Describe the issue It seems that it's a more conventional setup to have versions in the init files of python packages and have them being referenced in the setup.py files. What if we do not do this? Just inconvenient Related component(s) flytekit Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1594 [Core feature] FlyteIDL should generate protobuf files for R Issue created by kumare3 Motivation: Why do you think this is important? RProtobuf is a package that allows marshalling and unmarshalling protobuf structs to R objects. Doing so would make it possible to write a task natively in R Goal: What should the final outcome look like, ideally? community should be able to implement an R sdk for writing tasks if not complete workflows Describe alternatives you've considered ContainerNative tasks Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1592 [Core feature] Simple templating for default annotations/labels/env vars Issue created by frsann Motivation: Why do you think this is important? At the moment it's possible to add hard-coded annotations/labels/env vars to all pods or on a task level, but there is no middle ground. There is no easy way configure the default annotations to have project or domain specific keys/values. One way of achieving this is by allowing simple templating in the default annotations/labels/env vars. (Another way would be to allow defining project/domain specific annotations, but that might serve a different use case) Goal: What should the final outcome look like, ideally? In the propeller config, you'd be able to define something like:

default-annotations:
      - annotationKey1: annotationValue1_{PROJECT}_{DOMAIN}   # -> annotationValue1_flytesnacks_development
      - key_{project}: True   # -> key_flytesnacks: True

The idea would be to use predefined variable names (

PROJECT

,

DOMAIN

, `WORKFLOW`(?) ) that are anyway readily available when creating task pods. Describe alternatives you've considered The alternative would be to allow defining project/domain specific keys/values, which would allow more fine-grain control (maybe all projects don't need the key/value defined?) but at the same time would be less flexible with regards to including new projects. It basically serves a slightly different use case. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1587 [Docs] Document architecture and design of Flyte co-pilot Issue created by kumare3 Description Flyte co-pilot powers Raw containers and should be documented ☑︎ Are you sure this issue hasn't been raised already? ☑︎ Have you read the Code of Conduct? flyteorg/flyte

#1564 When running in a k8s cluster with mutating webhooks, tasks don't finish since the side car is still working [Core Feature] Issue created by NotMatthewGriffin Motivation: Why do you think this is important? Currently when running a flyte task in a kubernetes cluster that uses mutating webhooks to add sidecars to all pods, the task and associated pod will run until terminated in the flyte web console. Inside of the pod, the container actually running the flyte task has completed and exited with a good status but the sidecar container is still running. To work around this a user must specify the task in their workflow as a pod task like

@task(task_config=Pod(pod_spec=pod_spec, pimary_container_name="container_name"))

. This means that users need to know the configuration of the cluster ( like will any sidecars be added automatically) in order to successfully create/run a flyte workflow. Goal: What should the final outcome look like, ideally? Ideally a user of flyte wouldn't need to declare their task as a pod task in order to run it in a kubernetes cluster that uses mutating webhooks to add sidecars to pods. The user could declare their task like

@task

and it would run the flyte task pod until the flyte task container finishes then the rest of the workflow would be run. Describe alternatives you've considered The only other alternative I've considered is continuing to use the workaround of declaring all workflows/tasks as pod tasks. flyteorg/flyte

#1561 [Housekeeping] Test Flytectl &amp; Flytekit on Windows Issue created by samhita-alla Describe the Issue We haven't yet thoroughly tested Flytekit and Flytectl on a windows OS. Would love someone with a windows OS to test this out. What if we do not do this? Flyte on windows may not be foolproof. Related component flytekit, flytectl flyteorg/flyte

#1556 [Plugin]BlazingSQL and Rapids dataframe support Issue created by kumare3 Why would this plugin be helpful to the Flyte community It would be great to have a simple plugin that would allow users to simply write BlazingSQL queries in their workflows to manipulate data-frames on a GPU as part of their Flyte workflows. An example of writing a query from the User point of view could be

task = BlazingSQL(
     input_schema=FlyteSchema,
    outputs=...,
    query=""" SELECT count(*) FROM taxi GROUP BY year(key)""",
    use_gpus=False, # True by default
)

Type of Plugin ☑︎ Python/Java interface only plugin ☐ Web Service (e.g. AWS Sagemaker, GCP DataFlow, Qubole etc...) ☐ Kubernetes Operator (e.g. TfOperator, SparkOperator, FlinkK8sOperator, etc...) ☐ Customized Plugin using native kubernetes constructs ☐ Other flyteorg/flyte

#1547 [Flytectl Bug] Update check should not reports out-of-date for latest version Issue created by AdrianoKF Describe the bug

flytectl version

reports that a new release is available, even directly after

flytectl upgrade

$ flytectl upgrade 
You have already latest version of flytectl

$ flytectl version

 A new release of flytectl is available: 0.3.4 → v0.3.4

{
  "App": "flytectl",
  "Build": "8ba75a6",
  "Version": "0.3.4",
  "BuildTime": "2021-10-01 16:08:44.983660372 +0200 CEST m=+0.019362263"
}

Minor nitpick: the

flytectl upgrade

error message contains a typo (missing "the") It seems that the version number specified in the build (used as

stdlibversion.Version

in <cmd/version/version.go|`cmd/version/version.go`> incorrectly omits the

v

of the Git tag for the release. This leads to

IsVersionGreaterThan

in <pkg/util/util.go|`pkg/util/util.go`> incorrectly reporting that the latest Github version is greater than the local build version. I haven't fully grasped the release process, but it seems that the Github release action sets the version number without the

v

prefix in the Goreleaser action, which would explain the discrepancy. Expected behavior No newer release should be reported after upgrading to the latest version using

flytectl upgrade

. [Optional] Additional context To Reproduce 1. Upgrade to latest release using

flytectl upgrade

2. Perform update check through

flytectl version

flyteorg/flyte

#1546 [Flytectl Feature] Gracefully handle config files with non-standard extensions Issue created by AdrianoKF Describe the feature/command for flytectl We are trying to use

flytectl

inside a Gitlab CI pipeline. Gitlab supports supplying CI variables mounted as files, where the environment variable points to a file containing the value of the CI var - so naturally we tried setting

FLYTECTL_CONFIG

through this mechanism. However, this fails with the following error message when trying to execute any flytectl command:

Error:
Unsupported Config Type ""
ERRO[0000]
Unsupported Config Type ""

Apparently, this error message originates in the spf13/viper library used by flytectl through flytestdlib. The error is raised because viper tries to detect the type of the config file from its extension, which is unset in our CI environment (Gitlab names the file identical to the variable name, e.g. the

FLYTECTL_CONFIG

variable will be stored under

/builds/resources/repo_name.tmp/FLYTECTL_CONFIG

). The error is not specific to Gitlab CI, but can be reproduced by pointing

FLYTECTL_CONFIG

at any file without an extension, see below. Steps to reproduce

# Note that ./config is a valid YAML flytectl config file
$ export FLYTECTL_CONFIG=$PWD/config
$ cat $FLYTYCTL_CONFIG
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///localhost:30081
  insecure: true
logger:
  show-source: true
  level: 0
storage:
  connection:
    access-key: minio
    auth-type: accesskey
    disable-ssl: true
    endpoint: <http://10.32.16.105:30084>
    region: us-east-1
    secret-key: miniostorage
  type: minio
  container: "my-s3-bucket"
  enable-multicontainer: true

$ flytectl get projects
Error:
Unsupported Config Type ""
ERRO[0000]
Unsupported Config Type ""

$ flytectl version
 A new release of flytectl is available: 0.3.4 → v0.3.4

{
  "App": "flytectl",
  "Build": "8ba75a6",
  "Version": "0.3.4",
  "BuildTime": "2021-10-01 15:37:24.613781861 +0200 CEST m=+0.029412676"
}

Proposed fix Viper offers the

SetConfigName

and

SetConfigType

methods which could be used in flytestdlib instead of

SetConfigFile

if the config file is always assumed to be a YAML file. flyteorg/flyte

#1539 Add support for Multiple Architecture types Issue created by akhurana001 Motivation: Why do you think this is important? Multiple cloud providers (AWS etc.) have extensive support for ARM based instances now and we want to explore running Flyte workloads on ARM instances due to multiple benefits over x86. In general, the goal is to enable platform admins to configure multiple architecture types as well as Flyte users to select between them based on which one they want to use/image support etc. Goal: What should the final outcome look like, ideally? Platform admins be able to configure multiple architecture support within the same flyte installation and users being able to specify which one to execute on. [Optional] Propose: Link/Inline OR Additional context The goal is to enable Platform admins to maintain both ARM and x86 ASGs on each Kubernetes cluster and to intelligently route workloads to the right ASG based on which architecture the user image was built/tested/registered for. This can be done with adding taints to the ASGs and tolerations/node-selector to pods something Flyte already does for Interruptible enabled Tasks. However, for this use-case, it might be easier for users to tag the whole repo/sha with ARM rather than tagging individual tasks/nodes. I was thinking we can re-use labels support at Launch-Plan level to implement this: Users can set architecture label either at project level via flyte-config or directly at a launch-plan level which will then be routed correctly to the Flyte backend for those workflow versions/sha and corresponding tolerations and node-selector be added to those pods by FlytePropeller/plugins (similar to how Interruptible works). Let me know what your thoughts are on this. flyteorg/flyte

#1523 [Flytectl Feature] flytectl status should show env vars to be exported Issue created by kumare3 Describe the feature/command for flytectl if you use flytectl sandbox, as a user you may lose the information about how to use kubectl & flytectl to connect to flyte-sandbox. We should allows users to recover this using

flytectl sandbox status

Provide a possible output or ux example

$ flytectl sandbox status
...
To Connect to FlyteSandbox Kube cluster export:
...
To connect to FlyteSandbox using flytectl export:
...

flyteorg/flyte

#1520 Add options to configure SendGrid API Key Issue created by ariefrahmansyah Motivation: Why do you think this is important? While working on notifications using GCP and SendGrid (flyteorg/flyteadmin#259), I need to supply the SendGrid API Key. To achieve that, I configure SENDGRID_API_KEY via environment variables. I deployed Flyte using Helm, but since there's no way to provide environment variables via Helm, I need to manually add SENDGRID_API_KEY. Goal: What should the final outcome look like, ideally? To be able to properly configure SendGrid as the email sender. Describe alternatives you've considered Enable user to specify additional environment variables for flyteadmin deployment. flyteorg/flyte

#1509 [Core Feature][flytekit] Ability to track resource utilization and gpu utilization of a python task as a plugin Issue created by kumare3 Motivation: Why do you think this is important? When users are running tasks that use GPU's, CPUs and Memory, it might be useful to allow some meta-outputs that render this information Goal: What should the final outcome look like, ideally? There are many libraries like https://github.com/gpuopenanalytics/pynvml etc that can be used to collect stats for various resources and we could either stream or outputs these stats as a rendered PNG at the end of the run. The rendering can be done either in python or on the server and the outputs should be meta outputs. Describe alternatives you've considered Use other external libraries like Wandb etc [Optional] Propose: Link/Inline OR Additional context Write a new plugin

flytekit-resource-monitoring

that can load components that can output certain meta outputs. For this, we will need support for meta-outputs. flyteorg/flyte

#1496 [Core Feature] Temp Disk Volumes Issue created by vsbus Motivation: Why do you think this is important? We can specify mem/cpu request/limit for the tasks. It will also be good to have a similar feature for disk. Note, k8s v1.21 has beta for Generic Ephemeral Volumes and they seem to provide what is needed. Goal: What should the final outcome look like, ideally? Feature available for all flyte task types. A usage of this feature could look like this

@flytekit.task(
    retries=1,
    requests=flytekit.Resources(cpu="12", mem="32Gi"),
    limits=flytekit.Resources(mem="64Gi", cpu="12"),
    temp_disk_volumes=[flytekit.Volume(size="100G", mount_path="/tmpdisk_01")],
)
def foo():
  download_large_file_to("/tmpdisk_01/abc")

Describe alternatives you've considered Maybe flyte users can do it today with Sidecar task. But having a support of this parameter in every task would bring more convenience. [Optional] Propose: Link/Inline OR Additional context If you have ideas about the implementation please propose the change. If inline keep it short, if larger then you link to an external document. flyteorg/flyte

#1457 [Docs] Improve debugging documentation for Flytepropeller logs Issue created by kumare3 Describe the documentation update FlytePropeller has logs of the following form

{
  "json": {
    "exec_id": "grs2u5m2ns",
    "node": "n0",
    "ns": "flytesnacks-development",
    "res_ver": "137261245",
    "routine": "worker-2",
    "src": "handler.go:313",
    "tasktype": "python-task",
    "wf": "flytesnacks:development:<http://core.control_flow.dynamics.wf|core.control_flow.dynamics.wf>"
  },
  "level": "warning",
  "msg": "No plugin found for Handler-type [python-task], defaulting to [container]",
  "ts": "2021-09-10T15:53:28Z"
}

There should be a document that explains how to filter logs to get only the logs for one specific workflow / execution id etc flyteorg/flyte

#1467 [flytekit] List plugins automatically in pythonbuild.yml Issue created by wild-endeavor Leftover todo from flyteorg/flytekit#644 We should detect when a new plugin is added automatically. flyteorg/flyte

#1456 [Core Feature]Generic Schemas should detect the schema at runtime Issue created by kumare3 Motivation: Why do you think this is important? It is possible to use Schema's in Flyte, without individual columns. this may be desirable for cases in which the schema is not completely known ahead of time. But, it might be useful to automatically detect the schema and populate it in the runtime transport Schema layer. the Transport allows us to carry the schema type Goal: What should the final outcome look like, ideally? In case a pandas.DataFrame is used, Flytekit should be able to supply the schema at runtime. This can be visualized in Flyteconsole Describe alternatives you've considered None [Optional] Propose: Link/Inline OR Additional context Parquet files carry the schema inherently. Also it is possible to extract the schema from a pandas dataframe flyteorg/flyte

#1422 [Core Feature] Support pause and unpause in scheduler in case admin is down Issue created by pmahindrakar-oss Motivation: Why do you think this is important? Scheduler uses admin for sending workflow executions at the scheduled time. But in case of admin being down currently we have a retry mechanism which through configured number of steps using exponential backoff. Admin being down is a special case where all scheduled workflows will go in this retry loop. The proposal is to introduce a pause mechanism for the scheduler where all the existing schedules are paused and resumed when admin is healthy. We will use admins health api to determine this. When unpaused the scheduler will catch up on all the schedules till the pause time cc : @EngHabu @Ketan (kumare3) @evalsocket Goal: What should the final outcome look like, ideally? The scheduler pauses on admin down and unpauses when admin comes back up healthy and catches up on all the schedules. Describe alternatives you've considered The existing retry mechanism works but it unnecessarily causes all go routines looping in retries [Optional] Propose: Link/Inline OR Additional context If you have ideas about the implementation please propose the change. If inline keep it short, if larger then you link to an external document. flyteorg/flyte

#1420 [Core Feature] Implement recovery for flytekit remote Issue created by katrogan Motivation: Why do you think this is important? Now that recovery mode is implemented, see #1151 it would be helpful to interact with executions by recovering them in flytekit remote. Note, this issue only covers flytekit remote. Goal: What should the final outcome look like, ideally? It should be possible to recover executions using flytekit remote Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered. [Optional] Propose: Link/Inline OR Additional context If you have ideas about the implementation please propose the change. If inline keep it short, if larger then you link to an external document. flyteorg/flyte

#1417 Creating new spark_sql_task Issue created by yunhao-qing Why would this plugin be helpful to the Flyte community _spark_sql_task_ will allow users to submit a query that will run a Spark. It will not necessarily run on native Spark kubernetes operator. It will be similar structure to current _hive_task_ or _presto_task_. Type of Plugin ☑︎ Python/Java interface only plugin ☐ Web Service (e.g. AWS Sagemaker, GCP DataFlow, Qubole etc...) ☐ Kubernetes Operator (e.g. TfOperator, SparkOperator, FlinkK8sOperator, etc...) ☐ Customized Plugin using native kubernetes constructs ☐ Other Can you help us with the implementation? ☑︎ Yes ☐ No flyteorg/flyte