#2898 [BUG] [flyte-scheduler] Error handling can lead to very high backoff Issue created by kumare3 Describe the bug Flyte Scheduler uses backoffs to try and recover from errors in communicating with FlyteAdmin. But, the backoff logic can lead to extremely large back-offs. https://github.com/flyteorg/flyteadmin/blob/f0e4c890ee4e872d32541a6e9f02adbeb2656f47/scheduler/executor/executor_impl.go#L99 Expected behavior The back-off should be capped at a few miinutes and lower bound should be slightly higher not 3000ns. Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2901 [Housekeeping] Deleted CRON schedule shorthands Issue created by SmritiSatyanV Describe the issue Keep tabs on flytekit to support CRON schedule shorthand. Currently, flytekit doesn’t support this format but the scheduler does.Use the regular 5 field standard cron for now. Ensure that shorthands work with the upcoming flytekit versions What if we do not do this? Users won't be able to use CRON shorthands. Related component(s) No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2896 [Docs] Flyte Tensorflow cookbook example doesn't work on Macbooks with Silicon Chip Issue created by jevers-bc Description Trying to run the Tensorflow example from the Flyte cookbook results in an error for users with the Mac silicon chip. Tutorial - link Tensorflow can be installed on a user's local machine via

pip install tensorflow-macos

, but using base image for the tutorial (

tensorflow/tensorflow:latest-gpu

or

tensorflow/tensorflow:latest

) causes the following error when a workflow execution is attempted in a sandbox deployment:

The TensorFlow library was compiled to use AVX instructions, but these aren't available on your machine.
qemu: uncaught target signal 6 (Aborted) - core dumped

Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2895 [BUG] Dataclass becomes empty after running ray remote function Issue created by pingsutw Describe the bug Slack thread Failed to serialize dataclass because the value became empty for some reason after running ray remote function Expected behavior Successfully serialize the dataclass Additional context to reproduce

from dataclasses import dataclass
from typing import cast
from dataclasses_json import dataclass_json, DataClassJsonMixin
import ray


@dataclass_json
@dataclass
class Datum(object):
    x: int
    y: str


@ray.remote
def ray_create_datum(x: int, y: str):
    return Datum(x=x, y=y)


d = Datum(x=1, y='hello')

print("value", cast(DataClassJsonMixin, d).to_json()) # value {"x": 1, "y": "hello"}
ray.get(ray_create_datum.remote(x=1, y='hello'))
print("value", cast(DataClassJsonMixin, d).to_json()) # value {}

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2890 [FlyteCTL Feature] Remove support for storage flag Issue created by eapolinario Describe the feature/command for FlyteCTL We should drop the

storage

flag entirely now that past Flyte 1.0. In flyteorg/flytectl#296 we removed the need for users to define a

storage

stanza in their config files and in flyteorg/flytectl#319 we stopped inserting that stanza in the generated config files, however the flag is still present in the latest versions of the flytectl. Provide a possible output or UX example We have users confused about the support of that flag in flytectl (e.g. [1]). Since we no longer support we should drop it entirely. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2884 [BUG] Getting started guide, can't run on local cluster Issue created by tamis-laan Describe the bug I'm trying out Flyte locally running through the getting started guide: https://docs.flyte.org/en/latest/getting_started/index.html The code runs great using

pyflyte

but doesn't work properly when running on the local demo cluster. running:

> pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2

results in the error:

{"asctime": "2022-09-15 13:06:21,069", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"DNS resolution failed for localhost:30081: C-ares status is not ARES_SUCCESS qtype=A name=localhost is_balancer=0: DNS server returned answer with no data\"\n\tdebug_error_string = \"UNKNOWN:DNS resolution failed for localhost:30081: C-ares status is not ARES_SUCCESS qtype=A name=localhost is_balancer=0: DNS server returned answer with no data {created_time:\"2022-09-15T13:06:21.069160907+02:00\", grpc_status:14}\"\n>, sleeping 200ms and retrying"}
{"asctime": "2022-09-15 13:06:21,270", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"DNS resolution failed for localhost:30081: C-ares status is not ARES_SUCCESS qtype=A name=localhost is_balancer=0: DNS server returned answer with no data\"\n\tdebug_error_string = \"UNKNOWN:DNS resolution failed for localhost:30081: C-ares status is not ARES_SUCCESS qtype=A name=localhost is_balancer=0: DNS server returned answer with no data {created_time:\"2022-09-15T13:06:21.269916921+02:00\", grpc_status:14}\"\n>, sleeping 400ms and retrying"}
Traceback (most recent call last):
  File "/home/tux/.local/bin/pyflyte", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/tux/.local/lib/python3.10/site-packages/flytekit/clis/sdk_in_container/run.py", line 539, in _run
    remote_entity = remote.register_script(
  File "/home/tux/.local/lib/python3.10/site-packages/flytekit/remote/remote.py", line 596, in register_script
    upload_location, md5_bytes = fast_register_single_script(
  File "/home/tux/.local/lib/python3.10/site-packages/flytekit/tools/script_mode.py", line 113, in fast_register_single_script
    upload_location = create_upload_location_fn(content_md5=md5)
  File "/home/tux/.local/lib/python3.10/site-packages/flytekit/clients/friendly.py", line 998, in get_upload_signed_url
    return super(SynchronousFlyteClient, self).create_upload_location(
  File "/home/tux/.local/lib/python3.10/site-packages/flytekit/clients/raw.py", line 41, in handler
    return fn(*args, **kwargs)
  File "/home/tux/.local/lib/python3.10/site-packages/flytekit/clients/raw.py", line 854, in create_upload_location
    return self._dataproxy_stub.CreateUploadLocation(create_upload_location_request, metadata=self._metadata)
  File "/home/tux/.local/lib/python3.10/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/tux/.local/lib/python3.10/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "DNS resolution failed for localhost:30081: C-ares status is not ARES_SUCCESS qtype=A name=localhost is_balancer=0: DNS server returned answer with no data"
	debug_error_string = "UNKNOWN:DNS resolution failed for localhost:30081: C-ares status is not ARES_SUCCESS qtype=A name=localhost is_balancer=0: DNS server returned answer with no data {created_time:"2022-09-15T13:06:21.670827636+02:00", grpc_status:14}"

Expected behavior The example code should run on the local cluster I created using:

> flytectl demo start

Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2881 [Core feature] Flytekit execution models should expose more fields Issue created by katrogan Motivation: Why do you think this is important? Currently the execution models expose a subset of the underlying execution fields. We should expose the abort_metadata in the closure and reference execution in the metadata Goal: What should the final outcome look like, ideally? Flytekit Execution models should expose fields from the underlying protobuf definition from executions fetched using flytekit remote Describe alternatives you've considered Current implementation Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2880 [BUG] Integer type cannot be reserved when changing python dict type to struct_pb2 Issue created by highfly22 Describe the bug If I pass an argument as dict, then the argument can not be reserved the integer type. So I have to use YAML string directly.

>>> from flytekit.core.type_engine import DictTransformer
>>> DictTransformer.dict_to_generic_literal({'a' : 1})
scalar {
  generic {
    fields {
      key: "a"
      value {
        number_value: 1.0
      }
    }
  }
}

Expected behavior I expect the result of the following code is integer 1 not float 1.0. Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2879 [Housekeeping] Papermill example failure? Issue created by mounesi Describe the issue I have followed the steps to get the papermill extension working with the jupyter notebook. I have ran into 500 GET error. https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/flytekit_plugins/papermilltasks/index.html What if we do not do this? People who want to get a quick try and experience of Flyte will fail running this jupyter integration and may never come back to test it again. Related component(s) No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2875 flytecl register throws Error on Windows Issue created by sakthi-tech4th Hi, I'm trying to create flyte workflows in Windows and tried to follow the 'Getting Started' instructions. When I try to register the packaged files, I'm getting this error. Error: CreateFile /tmp: The system cannot find the file specified. Here is a screenshot: I believe this is because flyte is trying to create a folder using Linux conventions, in windows. I'm trying to deploy it locally using the sandbox environment as per the instructions So my questions are: 1. Does Flyte Support Windows 10? 2. If not, are there any quick patches to fix this issue? Kindly advise. Thanks flyteorg/flyte

#2874 [Housekeeping] [plugins] Remove Hive/Presto Issue created by wild-endeavor Describe the issue Remove Hive & Presto plugins What if we do not do this? They will continue to confuse potential contributors. Related component(s) flytekit, flyteplugins. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2868 [BUG] New flytectl version ignores storage settings from config file Issue created by fg91 Describe the bug flytectl

v0.6.12

ignores the storage settings, e.g. endpoint url, in the config file while

v0.5.8

does not.

admin:
  ...
logger:
   ...
storage:
  connection:
    endpoint: <http://minio.flyte.svc.cluster.local:9000>
    ...
  type: minio
  container: "my-s3-bucket"
  enable-multicontainer: true

(I have not tested whether other settings are ignored as well.) Expected behavior The new version should not ignore the storage settings. Additional context to reproduce I'm executing the following script in a container with either

v0.6.12

(latest) or `v0.5.8`:

pyflyte --pkgs flytesnacks.workflows package --fast --source . --image k3d-registry.localhost:5000/workflow:latest -f

cat > config.yaml <<'_EOF'
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///flyteadmin.flyte.svc.cluster.local:81
  authType: Pkce
  insecure: true
logger:
  show-source: true
  level: 0
storage:
  connection:
    access-key: minio
    auth-type: accesskey
    disable-ssl: true
    endpoint: <http://minio.flyte.svc.cluster.local:9000>
    region: us-east-1
    secret-key: miniostorage
  type: minio
  container: "my-s3-bucket"
  enable-multicontainer: true
_EOF

flytectl version
flytectl --config config.yaml register files --project flytesnacks --domain development --archive flyte-package.tgz --version {{inputs.parameters.version}}

I'm getting the following for `v0.6.12`:

{
  "App": "flytectl",
  "Build": "0a68a57",
  "Version": "0.6.12",
  "BuildTime": "2022-09-12 15:20:25.6728525 +0000 UTC m=+0.071785901"
}{
  "App": "controlPlane",
  "Build": "f10a7f2",
  "Version": "v1.1.28-2-gf10a7f2",
  "BuildTime": "2022-09-12 11:58:02.8187065 +0000 UTC m=+0.044260701"
Error: failed to upload source code from [/tmp/register4188038613/fastfc94cfb95c9d5602278cd12d2c5a08c3.tar.gz]. Error: Put "http://l
ocalhost:30084/my-s3-bucket/91/flytesnacks/development/3OHD3Z5W76BDAC2Q5XFRGTPLCY%3D%3D%3D%3D%3D%3D/fastfc94cfb95c9d5602278cd12d2c5a
08c3.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20220912%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220912T1
52031Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=e646c17183a38a5a4aa0c51d59a6ae7ba19977234399c761181
541758c8667c2": dial tcp [::1]:30084: connect: connection refused

Notice the

Error: Put "<http://localhost:30084/my-s3-bucket>

which is not the endpoint url specified in the config file. For

v0.5.8

it works as expected:

{
  "App": "flytectl",
  "Build": "ebda958",
  "Version": "0.5.8",
  "BuildTime": "2022-09-12 15:26:51.7031919 +0000 UTC m=+0.064945701"
}{
  "App": "controlPlane",
  "Build": "f10a7f2",
  "Version": "v1.1.28-2-gf10a7f2",
  "BuildTime": "2022-09-12 11:58:02.8187065 +0000 UTC m=+0.044260701"
} ---------------------------------------------------------------------------- --------- ------------------------------
| NAME (5)                                                                   | STATUS  | ADDITIONAL INFO              |
 ---------------------------------------------------------------------------- --------- ------------------------------
| /tmp/register1258794327/0_flytesnacks.workflows.workflow.prepare_data_1.pb | Success | Successfully registered file |
...

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2866 [Docs] Specify fluent-bit's requirement while pushing logs from Flyte to cloudwatch Issue created by SmritiSatyanV Description Amazon recommends using that https://aws.amazon.com/premiumsupport/knowledge-center/cloudwatch-stream-container-logs-eks/ for streaming logs from eks containers https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-EKS-logs.html If there are logs in cloudwatch, user needs to use fluent-bit to push logs from Flyte to Cloudwatch. Slack conversation flyteorg/flyte

#2863 [Core feature] Use https://github.com/Textualize/rich for pyflyte commands to help build great UX Issue created by kumare3 Motivation: Why do you think this is important? Sometimes errors are not easy to understand, sometimes stack traces are raised. Also tables and other things can be rendered much better with rich Goal: What should the final outcome look like, ideally? Beautiful rendering of the CLI Describe alternatives you've considered NA Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2859 [Core feature] Per-domain secrets mounting Issue created by slauth-io-tal Motivation: Why do you think this is important? When running a task within a certain domain (production/staging/development), it should be (security-)isolated from other domains. Meaning, secrets should be bound to a single domain and shouldn't be accessible to other domains. An example - while the same task in different domains might require a database secret, the tasks running in the development domain shouldn't have access to the production database. Nor should a task determine what secret to request/use based on its domain - secrets should be treated like an environment configuration. Goal: What should the final outcome look like, ideally? Tasks are able to request a secret, and during runtime the control plane determines which secret to mount according to the task domain. So, a task running in development and requesting the database secret will get a secret representing the development database. Describe alternatives you've considered Currently implementing this mechanism within the code of tasks. Preferably this'll be included in Flyte itself. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2853 [Docs] pyflyte register examples Issue created by SmritiSatyanV Description To improve searchability of

pyflyte register

, add more examples mentioning

pyflyte register

Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2832 [Core feature] Specifying TTL for flyte managed files Issue created by hajapy Motivation: Why do you think this is important? Data that is passed from tasks in flyte workflows is typically in the form of a FlyteFile, which abstracts away how that data is stored on a remote object store, like s3. Regulatory requirements can often impose a lifetime on data, and there is no obvious way to specify the lifetime of such files that are managed by Flyte. By supporting a specification like a time to live (TTL) at the Flyte abstraction layer, it would be possible as a user to apply these types of expirations to objects in a granular manner (only expire data that needs to be expired) in a cloud-agnostic way and ultimately enable users to meet their regulatory requirements. Goal: What should the final outcome look like, ideally? Ideally, when constructing a FlyteFile or FlyteDirectory, the user could pass in an optional TTL or expiry timestamp. Once that object gets created by a flyte workflow, the flyte backend would be aware of it and via a periodic job clean up the object(s) once they have expired. A record of the expiration action, obtainable through flyte's apis or visible somewhere in the flyteconsole, would assist in auditability and compliance efforts. Past workflows that are either re-run or otherwise inspected should handle the fact that some object(s) may be expired. Describe alternatives you've considered 1. Apply lifetime policies to objects in the flyte backend blob store. This would delete objects that are not actually needing to be deleted, so not optimal. 2. Apply lifetime policies to objects in the flyte backend object store in conjunction with some tag filtering whereby Flyte would still somehow know to tag certain objects but delegate the cleanup to the backend store. This is not as granular as the proposal and would need cloud-vendor specific implementations. One also needs to coordinate the bucket configuration with the Flyte configuration to get this to work. In both cases, Flyte may be unaware of those objects being deleted by the configured backend object store and lead to a suboptimal user experience interacting with past workflows where those objects were passed / output. Propose: Link/Inline OR Additional context Suggested in slack thread Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2821 [Docs] An example showing AWS EFS volume in eks cluster for ``ReadWriteMany`` Issue created by SmritiSatyanV Description Provide an example that illustrates how EFS volume can be used in eks cluster. Ketan: I think we should have an example of using Efs Reference Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2814 [BUG] flyteadmin make generate has different results between go 1.18 and 1.19 Issue created by andrewwdye Describe the bug Running

make generate

on a clean flyteadmin repo produces different results between go versions 1.18 and 1.19. This causes the Check Go Generate checker to fail and prevents using the latest version of go for development. Looks like this is related to

enumer

and alvaroloes/enumer#68. Expected behavior Running

make generate

on a clean flyteadmin repo with the latest version of go produces the same output as Check Go Generate. Additional context to reproduce 1. Install latest version of go (1.19) 2. From a clean flyteadmin repo, run

make generate

3. Run

git diff

diff --git a/auth/config/authorizationservertype_enumer.go b/auth/config/authorizationservertype_enumer.go
index a5c7dc2..f6e89a6 100644
--- a/auth/config/authorizationservertype_enumer.go
+++ b/auth/config/authorizationservertype_enumer.go
@@ -1,6 +1,5 @@
 // Code generated by "enumer --type=AuthorizationServerType --trimprefix=AuthorizationServerType -json"; DO NOT EDIT.

-//
 package config

 import (
diff --git a/auth/config/samesite_enumer.go b/auth/config/samesite_enumer.go
index af9bfdf..e42e58f 100644
--- a/auth/config/samesite_enumer.go
+++ b/auth/config/samesite_enumer.go
@@ -1,6 +1,5 @@
 // Code generated by "enumer --type=SameSite --trimprefix=SameSite -json"; DO NOT EDIT.

-//
 package config

 import (
diff --git a/pkg/runtime/interfaces/inlineeventdatapolicy_enumer.go b/pkg/runtime/interfaces/inlineeventdatapolicy_enumer.go
index 63ff94e..7c3895b 100644
--- a/pkg/runtime/interfaces/inlineeventdatapolicy_enumer.go
+++ b/pkg/runtime/interfaces/inlineeventdatapolicy_enumer.go
@@ -1,6 +1,5 @@
 // Code generated by "enumer -type=InlineEventDataPolicy -trimprefix=InlineEventDataPolicy"; DO NOT EDIT.

-//
 package interfaces

 import (

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2812 [Core feature] Explore FlyteAdmin slow SQL Issue created by katrogan Motivation: Why do you think this is important? A running Flyte deployment logs frequent slow SQL:

flyteadmin-794f779986-mqrs7 flyteadmin 
flyteadmin-794f779986-mqrs7 flyteadmin 2022/08/24 19:09:37 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/workflow_repo.go:79 SLOW SQL >= 200ms
flyteadmin-794f779986-mqrs7 flyteadmin [703.604ms] [rows:1] SELECT * FROM "workflows" WHERE project = 'flytesnacks' AND domain = 'development' AND name = 'core.control_flow.high_cpu_wf.single_integer_map_task' ORDER BY created_at desc LIMIT 1
flyteadmin-794f779986-mqrs7 flyteadmin 
flyteadmin-794f779986-mqrs7 flyteadmin 2022/08/24 19:09:37 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/workflow_repo.go:79 SLOW SQL >= 200ms
flyteadmin-794f779986-mqrs7 flyteadmin [720.675ms] [rows:1] SELECT * FROM "workflows" WHERE project = 'flytesnacks' AND domain = 'development' AND name = '<http://core.control_flow.dynamics.wf|core.control_flow.dynamics.wf>' ORDER BY created_at desc LIMIT 1
flyteadmin-794f779986-mqrs7 flyteadmin 
flyteadmin-794f779986-mqrs7 flyteadmin 2022/08/24 19:09:37 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/workflow_repo.go:79 SLOW SQL >= 200ms
flyteadmin-794f779986-mqrs7 flyteadmin [747.358ms] [rows:1] SELECT * FROM "workflows" WHERE project = 'flytesnacks' AND domain = 'development' AND name = 'core.control_flow.conditions.multiplier_2' ORDER BY created_at desc LIMIT 1
flyteadmin-794f779986-mqrs7 flyteadmin 
flyteadmin-794f779986-mqrs7 flyteadmin 2022/08/24 19:09:37 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/launch_plan_repo.go:133 SLOW SQL >= 200ms
flyteadmin-794f779986-mqrs7 flyteadmin [483.240ms] [rows:1] SELECT "launch_plans"."id","launch_plans"."created_at","launch_plans"."updated_at","launch_plans"."deleted_at","launch_plans"."project","launch_plans"."domain","launch_plans"."name","launch_plans"."version","launch_plans"."spec","launch_plans"."workflow_id","launch_plans"."closure","launch_plans"."state","launch_plans"."digest","launch_plans"."schedule_type" FROM "launch_plans" inner join workflows on launch_plans.workflow_id = workflows.id WHERE launch_plans.project = 'flytesnacks' AND launch_plans.domain = 'development' AND launch_plans.name = 'core.control_flow.chain_entities.chain_workflows_wf' LIMIT 1
flyteadmin-794f779986-mqrs7 flyteadmin 
flyteadmin-794f779986-mqrs7 flyteadmin 2022/08/24 19:09:37 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/launch_plan_repo.go:133 SLOW SQL >= 200ms
flyteadmin-794f779986-mqrs7 flyteadmin [485.314ms] [rows:1] SELECT "launch_plans"."id","launch_plans"."created_at","launch_plans"."updated_at","launch_plans"."deleted_at","launch_plans"."project","launch_plans"."domain","launch_plans"."name","launch_plans"."version","launch_plans"."spec","launch_plans"."workflow_id","launch_plans"."closure","launch_plans"."state","launch_plans"."digest","launch_plans"."schedule_type" FROM "launch_plans" inner join workflows on launch_plans.workflow_id = workflows.id WHERE launch_plans.project = 'flytesnacks' AND launch_plans.domain = 'development' AND launch_plans.name = 'core.control_flow.chain_tasks.chain_tasks_wf' LIMIT 1
flyteadmin-794f779986-mqrs7 flyteadmin 
flyteadmin-794f779986-mqrs7 flyteadmin 2022/08/24 19:09:37 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/launch_plan_repo.go:133 SLOW SQL >= 200ms
flyteadmin-794f779986-mqrs7 flyteadmin [406.597ms] [rows:1] SELECT "launch_plans"."id","launch_plans"."created_at","launch_plans"."updated_at","launch_plans"."deleted_at","launch_plans"."project","launch_plans"."domain","launch_plans"."name","launch_plans"."version","launch_plans"."spec","launch_plans"."workflow_id","launch_plans"."closure","launch_plans"."state","launch_plans"."digest","launch_plans"."schedule_type" FROM "launch_plans" inner join workflows on launch_plans.workflow_id = workflows.id WHERE launch_plans.project = 'flytesnacks' AND launch_plans.domain = 'development' AND launch_plans.name = '<http://control_flow.large_fanout_lp.wf|control_flow.large_fanout_lp.wf>' LIMIT 1

we should investigate indices or other performance optimizations to improve the query performance Goal: What should the final outcome look like, ideally? Fast queries to select workflows and launch plans Describe alternatives you've considered slower performance Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2811 [Plugin] Plotly Flyte deck renderer Issue created by cosmicBboy Flyte supports visualizing arbitrary artifacts in html form via Flyte Decks. Currenty the

flytekitplugins-deck-standard

contains a few renderers for dataframes, markdown, and box plots. The purpose of the issue is to add another renderer class for arbitrary plotly graphs. flyteorg/flyte

#2810 [Core feature] Flytekit plugin termination hook Issue created by hajapy Motivation: Why do you think this is important? As a flytekit plugin author, I would like to be able to define cleanup actions (in python) that should be run when a flyte task or workflow is aborted (or terminates unexpectedly). For example, this could be used to let the plugin cancel long-running operations in external services if a user aborts a workflow execution. Goal: What should the final outcome look like, ideally? The flytekit PythonTask interface could expose a new method (analogous to execute) that would be called upon termination of the task via an external signal. Describe alternatives you've considered An effort was made to use the stdlib's signal module to register handlers from the custom plugin module, but this didn't ultimately work (the signal seemed to be intercepted by pyflyte-execute). Propose: Link/Inline OR Additional context This suggestion was discussed in a flyte slack thread. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2727 [BUG] GCP Opta Deployment does not set roles and permissions Issue created by tszumowski Describe the bug When I run the opta GCP deployment steps, everything gets deployed including the service accounts. However, it did not apply the Roles or Permissions described here. I needed to manually follow those directions after running the Opta deployment in order for Flyte to work. Expected behavior Created GCP Service Accounts should have the permissions required to read/write from buckets as well as have the FlyteAdmin role should have the iam.serviceAccounts.signBlob applied. All permissions defined in the manual instructions should be in the deployment. Additional context to reproduce See this thread for details. To reproduce, deploy the Opta GCP configuration and check the service account permissions for FlyteAdmin. It should have signBlob shown above. I attached the logs from my run. opta_gcp_deploy_logs.txt Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2723 [BUG] Flytectl crashes on registration of workflow definitions coming from flytekit-java Issue created by sbrunk Describe the bug Flytectl 0.6.5, when trying to register serialized task/workflow definitions serialized with flytekit-java/jflyte, crashes with a segfault:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x1edd4ec]

goroutine 1 [running]:
<http://github.com/flyteorg/flytectl/cmd/register.hydrateIdentifier(0x0|github.com/flyteorg/flytectl/cmd/register.hydrateIdentifier(0x0>, {0x7ff7bfeff64c, 0x2}, 0x0)
        /home/runner/work/flytectl/flytectl/cmd/register/register_util.go:255 +0x2c
<http://github.com/flyteorg/flytectl/cmd/register.hydrateSpec({0x2f304c8|github.com/flyteorg/flytectl/cmd/register.hydrateSpec({0x2f304c8>?, 0xc000a6d7a0?}, {0x0, 0x0}, {{0x7ff7bfeff64c, 0x2}, 0x0, 0x0, 0x0, {0x0, ...}, ...})
        /home/runner/work/flytectl/flytectl/cmd/register/register_util.go:462 +0xdd
<http://github.com/flyteorg/flytectl/cmd/register.registerFile({0x2f3dc28|github.com/flyteorg/flytectl/cmd/register.registerFile({0x2f3dc28>, 0xc000120000}, {0x7ff7bfeff5e7, _}, {_, _, _}, {0x0, {0x0, 0x0}, ...}, ...)
        /home/runner/work/flytectl/flytectl/cmd/register/register_util.go:594 +0x488
<http://github.com/flyteorg/flytectl/cmd/register.Register({0x2f3dc28|github.com/flyteorg/flytectl/cmd/register.Register({0x2f3dc28>, 0xc000120000}, {0xc0003b3810?, 0x0?, 0x0?}, 0x3f20520, {0x0, {0x0, 0x0}, {0x0, ...}, ...})
        /home/runner/work/flytectl/flytectl/cmd/register/files.go:160 +0x6ca
<http://github.com/flyteorg/flytectl/cmd/register.registerFromFilesFunc(|github.com/flyteorg/flytectl/cmd/register.registerFromFilesFunc(>{0x2f3dc28, 0xc000120000}, {0xc0003b3810, 0x1, 0x7}, {0x0, {0x0, 0x0}, {0x0, 0x0}, ...})
        /home/runner/work/flytectl/flytectl/cmd/register/files.go:118 +0xcd
<http://github.com/flyteorg/flytectl/cmd/core.generateCommandFunc.func1(0xc000319180|github.com/flyteorg/flytectl/cmd/core.generateCommandFunc.func1(0xc000319180>?, {0xc0003b3810, 0x1, 0x7})
        /home/runner/work/flytectl/flytectl/cmd/core/cmd.go:65 +0x625
<http://github.com/spf13/cobra.(*Command).execute(0xc000319180|github.com/spf13/cobra.(*Command).execute(0xc000319180>, {0xc0003b37a0, 0x7, 0x7})
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:852 +0x67c
<http://github.com/spf13/cobra.(*Command).ExecuteC(0xc00051db80)|github.com/spf13/cobra.(*Command).ExecuteC(0xc00051db80)>
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:960 +0x39c
<http://github.com/spf13/cobra.(*Command).Execute(...)|github.com/spf13/cobra.(*Command).Execute(...)>
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:897
<http://github.com/flyteorg/flytectl/cmd.ExecuteCmd()|github.com/flyteorg/flytectl/cmd.ExecuteCmd()>
        /home/runner/work/flytectl/flytectl/cmd/root.go:137 +0x1e
main.main()
        /home/runner/work/flytectl/flytectl/main.go:12 +0x1d

Expected behavior flytectl should not crash. It should succeed or produce a useful error message if the input is invalid. Additional context to reproduce

git clone <https://github.com/spotify/flytekit-java.git>
cd flytekit-java
git checkout 0.3.15 # we need to use the previous release as there is another issue currently with 0.3.16
mvn package -DskipTests=true

flytectl sandbox start # we need a sandbox because jflyte wants to upload jars even on serialize currently
cat <<EOF > .env.local
FLYTE_PLATFORM_URL=localhost:30081
FLYTE_AWS_ENDPOINT=<http://localhost:30084>
FLYTE_AWS_ACCESS_KEY_ID=minio
FLYTE_AWS_SECRET_ACCESS_KEY=miniostorage
FLYTE_STAGING_LOCATION=<s3://my-s3-bucket>
FLYTE_PLATFORM_INSECURE=True
EOF


mkdir  _pb_output
scripts/jflyte serialize workflows -cp=flytekit-examples-scala/target/lib -f  _pb_output

flytectl register files  _pb_output/* -d development  -p flytesnacks  --version v1 # segfault

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2707 [Core feature] Speed up Deployment of a cluster - esp. for getting started or testing Issue created by brucearctor Motivation: Why do you think this is important? Takes ~30 minutes to go through the various stages. Explore, and ideally find ways, to get that time much less. Goal: What should the final outcome look like, ideally? Faster cluster [ EKS ] and Flyte specific starting times. Describe alternatives you've considered Not doing anything. Propose: Link/Inline OR Additional context Imagining this will either involve finding opta optimizations, or dropping into Terraform to reduce the number of stages, while being explicit about

depends_on

. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2694 [Docs] Publish Code Style Guide Issue created by brucearctor on <!date^1657815975^{date_short}|2022-07-14T16:26:15Z> Description It is great for a project to have consistent code formatting/style. This is often accomplished by a linting, and as part of pre-commit tests. But, it would also be helpful if a styleguide were published to make the choices/opinions explicit. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2681 [Docs] Guide on how to write data persistence plugins Issue created by cosmicBboy Description We currently support S3, GCS, and Local FS data persistence for when Flyte has to serialize data that's passed from one task to another, we should write a guide on how to write additional ones for e.g. Azure's blob store, etc. This issue breaks up the tasks described in #1541 Example: https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-data-fsspec Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2680 [Docs] Guide on how to write SQL plugins Issue created by cosmicBboy Description We have several SQL-related plugins like the SQL, Snowflake, and BigQuery plugins, but we need an extension guide as to how to write flytekit operators for other SQL-like packages. This issue breaks up the tasks described in #1541 Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2679 [Docs] Guide on how to write Flytekit shim plugins for Backend plugins Issue created by cosmicBboy Description Currently, we have docs for how to write backend plugins but don't have any guides for how to write a flytekit shim plugin that allows flytekit task authors to use that backend plugin. An example of this is the Sagemaker Training plugin, which requires both. This issue breaks up the tasks described in #1541 Context: Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte