#2652 [Docs] Docs on how Propeller ensures a task is executed exactly once Issue created by sonjaer Description I was curious how Propeller ensures a task is executed exactly once? I was curious since we have non idempotent tasks that we wouldn't want to run twice. Is there any instance when this could not be true? When deploying a new Propeller or error writing to etcd? Is this something that could be added to the Propeller Architecture docs here Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2758 [Docs] Describe with example how with_overrides can be used with dynamic workflows Issue created by SmritiSatyanV Description It is possible to mix @flytekit.dynamic & with_overrides. Pass a resource configuration object to a workflow at runtime and have the @dynamic step specify overrides at runtime. Document this with an example Github issue Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2754 [Docs] Document how to mount and dynamically create persistent volume Issue created by SmritiSatyanV Description Slack conversation

User_1: Hello flyte team. I have a question regarding using persistent volumes (PVs) and persistent volume claims (PVCs) with flyte tasks and workflows. More specifically, I’m interested in using an AWS efs volume in my eks cluster as a shared mount for ReadWriteMany. I have looked at [this example](<https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/pod/pod.html>) and see the V1Volume being passed into the pod spec. I’m looking through [these docs](<https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1Volume.md>) and see that you can create a V1Volume from a persistent_volume_claim . So in theory, I can just create a PV and PVC in my cluster, include them in my pod spec, and attach the spec to a flyte task. But I noticed that PVCs are namespace specific and flyte uses the project-domain namespace for tasks/workflows that are executing. Two questions.
Are PVCs the right solution here and if so, how can I dynamically create PVCs for my project-domain s? Is this something flyte could be configured to do for us or would we  be responsible for ensuring any referenced PVCs and PVs exist.
What other options are available for mounting shared persistent volumes to my flyte tasks?

User_2: I have considered exactly the same problem in my team.
here are some solution I do.
you can create pvc per namespace using IaC platform like Terraform, Pulumi.
if you are using IaC , then you don't have to create each PVC per namespace manually.
or you should make shell script to build all PVC per namespace at once.
how about using other path in V1Volume? you can use NFS or hostpath to access some specific volume. these are also be provided in V1Volume (kubernetes client, indeed)

Yuvraj [Response]: You guys can use cluster_resource_manager , For creating PVC for each project-domain namespace like this <https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/values-eks.yaml#L360>
[@Mike Zhong](<https://flyte-org.slack.com/team/U03BXNGV65A>)
 There are two ways to mount volume in your flyte task,
First is using [pod plugin](<https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/pod/pod.html>)
Second is propeller’s pod template  <https://docs.flyte.org/en/latest/deployment/cluster_config/general.html#using-default-k8s-podtemplates>

User_1: The cluster_resource_manager block is optional, but if we define namespace PVCs there, we would need to add them as new projects are created and redeploy to keep things updated. I was able to seamlessly get EFS working with flyte tasks so that was nice, just trying to figure out the best way to grant and manage access to the PV from other projects
other flyte projects (-p). Since the namespaces are {project}-{domain} and PVCs exist per namespace, any project that wants to use an efs backed PV would need the PVC created for its namespace. Please correct me if this is not correct

Ketan [Response]: That is true. But, if you are using EFS just create PVC in every namespace and then use podtemplates to mount them.
Now to provision them you can use ClusterResourceManager, or
you can use flytectl (gitops) style to create namesapces and manage resources in that namespaces through your regular CI/CD tooling. and you can have a centralized way to onboard onto Flyte - Creating a project has to go through a central way

Document : 1. How to mount shared persistent volumes to Flyte tasks 2. How to dynamically create Persistent Volume Claims (PVCs) for project-domains flyteorg/flyte

#2751 [Core feature] Create OSS Cloud Environments Issue created by brucearctor Motivation: Why do you think this is important? Community/Project could use Cloud Resources, currently - as I understand - these things are all Union-internal. Goal: What should the final outcome look like, ideally? things running in flyte/open-source environments with permissions to condoned community members Describe alternatives you've considered keeping things the way they are. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2731 [BUG] [flyteconsole] Dynamic task information not fully parsed Issue created by wild-endeavor Describe the bug When running a dynamic task like so clicking on the underlying generated task (n0-0-dn0), the pane that comes up on the right is for the parent dynamic task, not the underlying child task. The frontend renders the graph correctly though and when you click on

my_task

here, the pane that comes up is • missing the Task tab • missing name and type and you just the the gray updating bar Misc Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2729 [Docs] Provide an option to deploy to GCP via Opta without requiring a registered domain Issue created by tszumowski Description Background Ref: This slack thread. Some users may wish to have a simple deployment-to-GCP option, but do not currently have a registered domain they wish to bind to Flyte. @jeevb provided a sample Envoy configuration that allows one to port-forward a single port from Kubernetes and have access to the UI without exposing to a domain. The suggestion is to add this Envoy option to the GCP deployment docs. Files The following was provided by @jeevb (with some simplifications in the config): envoy.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flyte-proxy
  labels:
    app: flyte-proxy
spec:
  selector:
    matchLabels:
      app: flyte-proxy
  template:
    metadata:
      labels:
        app: flyte-proxy
    spec:
      containers:
        - name: proxy
          image: envoyproxy/envoy:v1.21.1
          args:
            - envoy
            - -c /etc/envoy/config.yaml
          ports:
            - name: http
              containerPort: 8000
          volumeMounts:
            - name: config-volume
              mountPath: /etc/envoy
      volumes:
        - name: config-volume
          configMap:
            name: flyte-proxy-config

config.yaml

admin:
  access_log_path: /dev/null
  address:
    socket_address:
      address: 127.0.0.1
      port_value: 9901
static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 8000
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": <http://type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager|type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager>
          stat_prefix: ingress_http
          codec_type: AUTO
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              domains:
              - "*"
              routes:
              - match:
                  path: "/"
                redirect:
                  path_redirect: "/console/"
              - match:
                  prefix: "/console"
                route:
                  cluster: flyteconsole
              - match:
                  prefix: "/__webpack_hmr"
                route:
                  cluster: flyteconsole
              - match:
                  prefix: "/api"
                route:
                  cluster: flyteadmin
              - match:
                  prefix: "/healthcheck"
                route:
                  cluster: flyteadmin
              - match:
                  prefix: "/v1"
                route:
                  cluster: flyteadmin
              - match:
                  prefix: "/login"
                route:
                  cluster: flyteadmin
              - match:
                  prefix: "/me"
                route:
                  cluster: flyteadmin
              - match:
                  prefix: "/flyteidl.service.AdminService"
                route:
                  cluster: flyteadmin_grpc
              - match:
                  prefix: "/openapi"
                route:
                  cluster: flyteadmin_redoc
          http_filters:
          - name: envoy.filters.http.router
  clusters:
  - name: flyteconsole
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: flyteconsole
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: flyteconsole
                port_value: 80
  - name: flyteadmin
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: flyteadmin
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: flyteadmin
                port_value: 80
  - name: flyteadmin_grpc
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    http2_protocol_options: {}
    load_assignment:
      cluster_name: flyteadmin_grpc
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: flyteadmin
                port_value: 81
  - name: flyteadmin_redoc
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: flyteadmin_redoc
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: flyteadmin
                port_value: 87

Directions 1. Create configmap in flyte namespace:

kubectl -n flyte create configmap flyte-proxy-config --from-file config.yaml

2. Apply the spec:

kubect -n flyte apply -f envoy.yaml

3. Port Forward:

kubectl -n flyte port-forward deployment/flyte-proxy 30080:8000

4. Update

~/.flyte/config.yaml

to point to `localhost:8000" Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2646 [Core feature] Easily see all workflow using a particular task Issue created by sonjaer Motivation: Why do you think this is important? [Nothing super important for us or a big use case yet, so feel free to ignore if you don't think it is valuable] It would be nice to see what workflows are using a particular task. Today we rewrote a task to replace an old task and wanted to remove the old task code completely. Before we wanted to verify if any workflows were using this task remotely. If that was the case, we might have asked the owners of the workflows to migrate to the new one or we would have kept the code in case there were any future changes needed. Goal: What should the final outcome look like, ideally? This or similar type of data could probably be useful for different use-cases, but I am not entirely sure what would be a final outcome. Describe alternatives you've considered I could get this probably by making different API calls Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2613 [Housekeeping] Create github workflows in Cookie Cutter template Issue created by EngHabu Describe the issue Cookie cutter template: https://github.com/flyteorg/flytekit-python-template Comes with a Dockerfile, scripts and configs but one thing it lacks is demonstrating best practices around iteration/development (CI/CD) as described here. I think we should add github workflows as follows: 1. PullRequest: a. Build Dockerfile b. Start flytectl demo cluster c. Register against it 2. Promote to Domain: a. Take the desired domain as an input (e.g. Staging/Production), a commit sha/tag, an optional URL of the org b. Build Dockerfile c. Start flytectl demo cluster (if org url is not provided) d. Register against it We should add instructions to advise people to: 1. Configure the Build Dockerfile step to push to their favorite Registry 2. Update the workflow code to default to their org if that's provided (or maybe make that an optional cookie cutter input?) What if we do not do this? Need to explain how users can build this using flyte's provided actions/tools Related component(s) No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2608 [UI Feature] Add links to github functions to be able to check the source code Issue created by stephen37 Motivation: Why do you think this is important? We have different users using Flyte and it would be nice to have a link to the code that is used to run the workflow/ tasks directly, it would make it easier for us when they need help. That would allow people to avoid having to look for the code on Github, Gitlab, etc and would make the whole process more efficient. I like the way Airflow does it as they simply show you the code directly in the UI but I think it is probably easier to add a link instead. Slack discussion: https://flyte-org.slack.com/archives/C02HARLSW30/p1651850298046559 Goal: What should the final outcome look like, ideally? When checking an execution, it would be nice to have a link where the source code is, maybe next to the "View Inputs & Outputs" would be nice. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2605 [Core feature] Improved and detailed error messages on JWT validation failures Issue created by katrogan Motivation: Why do you think this is important? When flyteadmin fails JWT token validation, the request fails with

Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken

but this is ambiguous. For example, if the audience is misconfigured, flyteadmin logs

"Failed to parse Access Token from context. Will attempt to find IDToken. Error: invalid audience

which is far more useful for understanding misconfigured deployments and diagnosing what went wrong Goal: What should the final outcome look like, ideally? Flyteadmin auth failures should include more details in error message responses Describe alternatives you've considered N/A Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2588 [Docs] correction for a command: pyflyte run Issue created by drleowang In https://docs.flyte.org/en/latest/getting_started/index.html There is a correction for a command: pyflyte run example.py wf --n 500 --mean 42 --sigma 2 pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2 should be pyflyte run example.py:wf --n 500 --mean 42 --sigma 2 pyflyte run --remote example.py:wf --n 500 --mean 42 --sigma 2 Thanks Leo flyteorg/flyte

#2569 [Plugin] TypeTransformer for TensorFlow tensor Issue created by samhita-alla We need to support

tf.tensor

as a native Flyte type. API reference for serialization and deserialization of tensors: • Serialization: https://www.tensorflow.org/api_docs/python/tf/io/serialize_tensor • Deserialization: https://www.tensorflow.org/api_docs/python/tf/io/parse_tensor More on TypeTransfomers can be found here. Related PR that adds PyTorch tensor and module as Flyte types: flyteorg/flytekit#1032 flyteorg/flyte

#2568 [Core feature] [Plugin] EMR Serverless spark support Issue created by kumare3 Motivation: Why do you think this is important? EMR Serverless spark for EKS is now generally available. https://aws.amazon.com/blogs/aws/new-amazon-emr-on-amazon-elastic-kubernetes-service-eks/ We can use this as an alternate plugin for spark. Goal: What should the final outcome look like, ideally? Spark plugin for K8s can simply use EMR serverless spark Describe alternatives you've considered K8s spark already exists in flyte Propose: Link/Inline OR Additional context NA Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2567 [Docs] Extend docs on how to use gpu nodes Issue created by katrogan Description https://docs.flyte.org/projects/cookbook/en/latest/auto/deployment/configure_use_gpus.html#sphx-glr-auto-deployment-configure-use-gpus-py should also include guidance to make sure users add a corresponding taint to their gpu nodes Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2559 [BUG] flyteadmin should not interfere with the format of client secrets Issue created by eapolinario Describe the bug Currently flyteadmin removes newlines from user-provided secrets. Expected behavior flyteadmin should load the contents of the secret file as it's provided, as having this extra cleaning step introduces potential divergences in how client secrets are used in other parts of the system, including client SDKs. At the very least we should detect the presence of trailing newlines and warn the user that having such characters there might indicate that this is not what they wanted in the first place. Additional context to reproduce Any client secret that contains a trailing newline triggers this behavior. Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2558 [Core feature] Expose task execution ids to client SDKs Issue created by eapolinario Motivation: Why do you think this is important? We should expose unique task execution names. There are some use cases that would benefit from uniquely identifying a specific task execution, including the possibility of using such unique ids to link external objects to specific task executions. Goal: What should the final outcome look like, ideally? The unique task id should be exposed to clients as an environment variable and each client sdk would access it at runtime similarly to how we expose the workflow execution id. The unique guarantees of such ids would follow the same as the workflow execution ids, in other words, they would be guaranteed to be unique in the context of a project and domain. Describe alternatives you've considered We could not do this and instead ask users to leverage other source of entropy to guarantee unique ids per task ids. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2547 [Docs] Document FlyteAdmin DB migration best practices Issue created by katrogan Description Somewhere within the contributors guide we should call out that since FlyteAdmin is backed by RBDMS special considerations must be given to model migrations. Specifically that: • they should always be additive • they should never alter existing column types Furthermore we should have upgrade best practices to document blue/green deploys to help users avoid downtime for potentially expensive migrations when upgrading flyte admin releases Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2540 [Housekeeping] [flytekit] Streamline module loading code Issue created by wild-endeavor TL;DR With all the work behind us in

pyflyte package

,

run

, and

register

, along with the legacy

serialize

, the pyflyte code could probably use some cleanup. Details This ticket includes but shouldn't be construed as to be limited to the following: • Use the same module loading code between the

run

and

register

commands. Since

run

has to first

register

, and since it also loads when listing commands in a module, it makes sense to reuse the same code. • There's a lot helper code in the

flytekit/tools

folder, and kind organized poorly. Attempt to streamline this. • Figure out proper handling of the

--pkgs

switch that exists at the top level of

pyflyte

, in the

register

command. Currently it just raises an error. Misc Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2523 [Docs] Getting started flytectl demo start for other operating systems Issue created by kdgosik In the Getting Started under creating a demo flyte cluster (https://docs.flyte.org/en/latest/getting_started/index.html#creating-a-demo-flyte-cluster) for other operating systems, I am having trouble getting it running. I was able to install flytectl via the curl command. I have to run

sudo flytectl demo start

to start the cluster but at the end, it tells me the connection is being refused. I saw the note about KUBECONFIG and FLTYECTL_CONFIG, but not sure if this will fix the issue or how or where to set these variables. Amy help would be greatly appreciated! I'm on an ubuntu laptop. flyteorg/flyte

#2520 [Update Request] Publish customised Sphinx extension to pypi to avoid installing main git repository Issue created by SmritiSatyanV Description Publish own sphinx extension to pypi so that we’re not installing the main branch of the git repo. We currently main a fork of furo repository which needs to be manually updated and installed via git. The build process is manual and lacks a pypi distribution for our fork. Slack ref: https://unionai.slack.com/archives/C02CDQQBQ2H/p1653054902643929?thread_ts=1653019414.762219&cid=C02CDQQBQ2H Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2519 [Update Request] Use updated furo package and create customized sphinx extension Issue created by SmritiSatyanV Description Depend on the main furo package Create own sphinx extension that extends furo and customizes the html/javascript/css based on our requirements Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2515 [BUG] Order of keys from DictTransformer is non-deterministic Issue created by ggydush-fn Describe the bug When using a dictionary with Flyte, the order of the keys is not deterministic. Python dictionary ordering is something that would be awesome to guarantee, as it was surprising to me when the order or the keys changed on Flyte. Expected behavior Maintain order of keys when using dictionaries on Flyte. Additional context to reproduce This example will fail. The

lit

and

transformed_lit

variables look identical, but are not after converting to Python value.

import string
from flytekit.core.context_manager import FlyteContext
from flytekit.core.type_engine import DictTransformer, TypeEngine
from flytekit.models import literals

def simulate_dict_transform(data: dict, python_type: type) -> dict:
    t = DictTransformer()
    ctx = FlyteContext.current_context()
    lit = t.to_literal(ctx, data, python_type, TypeEngine.to_literal_type(python_type))
    transformed_lit = literals.Literal.from_flyte_idl(lit.to_flyte_idl())
    return t.to_python_value(ctx, transformed_lit, python_type)

data = {k: v for v, k in enumerate(string.ascii_lowercase)}
transformed = simulate_dict_transform(data, dict[str, int])
print(data.keys())
print(transformed.keys())
assert list(data.keys()) == list(transformed.keys())

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2514 [Docs] Better coverage of pyflyte's local behavior Issue created by marc-union Description We need deeper coverage in the documentation for pyflyte's behavior for localized behavior. From a flyte-org thread in Slack:

The Getting Started example executes a workflow locally using the default Docker image in the sandbox server, right? But I thought that the code was required to be inside the container so that Flyte could run it. Does pyflyte run an implicit fast serialization? If not, what is it doing?

@Ketan (kumare3) 's (abridged) answer:

pyflyte

talks now (using flyteremote) to Flyteadmin to get a signedURL which is bound to the compressed archive of the code.

This is where

pyflyte

uploads the code (we wanted to make things easier for the user)

You can supply an alternate image, if not, it uses a default image with only flytekit and pandas/numpy installed in it.

References: https://flyte-org.slack.com/archives/CNMKCU6FR/p1652968591652399 https://docs.flyte.org/en/latest/getting_started/index.html#executing-workflows-on-a-flyte-cluster Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2513 [Docs] Add information about behaviors of resource defaults and limits Issue created by marc-union Description We need clearer documentation about how Flyte sets resource defaults and limits. From @katrogan :

Platform defaults are the defaults when no task level requests/limits are set.

If only one task level request/limit is set AND that request is under the platform limit both values will be set to whichever value was set in the task.

If only one task level request/limit is set AND that request is over the platform limit the unset value is set to the platform limit.

For full details, see the Slack thread below. References: https://flyte-org.slack.com/archives/C03CY9S9MFE/p1652970945750369 https://docs.flyte.org/en/latest/concepts/admin.html#task-resource-defaults Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2511 [UI Feature] Console should report created_at time for executions in 'UNKOWN' Issue created by katrogan Motivation: Why do you think this is important? When filtering executions stuck in 'UNKNOWN' on the project landing page there's no indication of how long the executions have been unknown or when they were first created. Since we have created_at in the executions proto it would be helpful context to surface that Goal: What should the final outcome look like, ideally? Have some indication of how long an execution has been in UNKNOWN. Propose: Link/Inline OR Additional context Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2506 [BUG] [flytectl] segfault when config file is missing Issue created by wild-endeavor Describe the bug flytectl segfaults apparently when the config file is missing. User reported when creating and listing projects. Issue went away when we created a sandbox style config.yaml in the .flyte directory.

INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xfef5a3]
​
goroutine 1 [running]:
<http://github.com/flyteorg/flytectl/cmd/core.CommandContext.AdminClient(...)|github.com/flyteorg/flytectl/cmd/core.CommandContext.AdminClient(...)>
        /home/runner/work/flytectl/flytectl/cmd/core/cmd_ctx.go:48
<http://github.com/flyteorg/flytectl/cmd/create.createProjectsCommand({0x23461e8|github.com/flyteorg/flytectl/cmd/create.createProjectsCommand({0x23461e8>, 0xc000050110}, {0x1ffffffffffffff?, 0x39?, 0x8?}, {0x0, {0x0, 0x0}, {0x0, 0x0}, ...})
        /home/runner/work/flytectl/flytectl/cmd/create/project.go:62 +0xe3
<http://github.com/flyteorg/flytectl/cmd/core.generateCommandFunc.func1(0xc000454c80|github.com/flyteorg/flytectl/cmd/core.generateCommandFunc.func1(0xc000454c80>?, {0xc00027e800, 0x0, 0x8})
        /home/runner/work/flytectl/flytectl/cmd/core/cmd.go:65 +0x625
<http://github.com/spf13/cobra.(*Command).execute(0xc000454c80|github.com/spf13/cobra.(*Command).execute(0xc000454c80>, {0xc00027e780, 0x8, 0x8})
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:852 +0x67c
<http://github.com/spf13/cobra.(*Command).ExecuteC(0xc00055b900)|github.com/spf13/cobra.(*Command).ExecuteC(0xc00055b900)>
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:960 +0x39c
<http://github.com/spf13/cobra.(*Command).Execute(...)|github.com/spf13/cobra.(*Command).Execute(...)>
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:897
<http://github.com/flyteorg/flytectl/cmd.ExecuteCmd()|github.com/flyteorg/flytectl/cmd.ExecuteCmd()>
        /home/runner/work/flytectl/flytectl/cmd/root.go:134 +0x1e
main.main()
        /home/runner/work/flytectl/flytectl/main.go:12 +0x1d

{
 "App": "flytectl",
 "Build": "c06589e",
 "Version": "0.5.25",
 "BuildTime": "2022-05-17 18:15:50.341414608 -0400 EDT m=+0.024193929"
}{
 "App": "controlPlane",
 "Build": "072ce74",
 "Version": "v1.1.2",
 "BuildTime": "2022-05-16 16:49:17.16516979 +0000 UTC m=+0.185367550"
}

Expected behavior A nice error message telling the user to run

config init

maybe? Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2505 [Core feature] Add ability to await first task of pyflyte run workflow Issue created by ggydush-fn Motivation: Why do you think this is important? Pyflyte run is an amazing improvement, but users have to leave the context of their terminal to check Flyte workflow run, or run flytectl commands repetitively to check the status. A design decision we've made, is to make the first task of a workflow lightweight (spec validation), and if this succeeds, a larger subworkflow/task will be executed. It may be useful to add the ability to wait for this first task to succeed before exiting pyflyte's CLI. Goal: What should the final outcome look like, ideally? I think 2 features to pyflyte run could be useful: 1. Print out flytectl command that can be used to monitor the workflow directly from CLI 2. Add an optional

--await-first-task

parameter that would wait for the first task to succeed before exiting the CLI. An indicator showing that the job was successful pushed to remote and is in running state would also be useful. For #2, I use something like this currently, and would love for this functionality to be added to flyte remote!

import time

from flytekit.clients.friendly import SynchronousFlyteClient
from flytekit.models.core.execution import NodeExecutionPhase
from flytekit.models.node_execution import NodeExecution
from flytekit.remote.executions import FlyteWorkflowExecution

from flyte_common.remote.remote import get_flyte_client
from flyte_common.remote.ui.status import status


class NodeMonitor:
    """Class for monitoring nodes in a workflow execution."""

    client: SynchronousFlyteClient = get_flyte_client()

    def __init__(self, workflow_execution: FlyteWorkflowExecution):
        self.workflow_execution = workflow_execution

    @status(message="Validating spec on Flyte")
    def wait_for_first_node_to_complete(self) -> None:
        """Wait for first node to complete before returning.

        Note: this is currently an experimental feature we've added to help validate
        specs remotely and notify user of any spec errors. It is recommended to have a
        lightweight first task so that your workflow will fail fast if there are problems
        with the spec.
        """
        first_node = self.get_first_node()
        while True:
            node = self.client.get_node_execution(first_node.id)
            status_string = NodeExecutionPhase.enum_to_string(node.closure.phase)
            if status_string in ["FAILED", "FAILING", "ABORTED", "TIMED_OUT"]:
                raise ValueError(
                    f"Spec validation failed. {node.closure.error.message}"
                )
            if status_string in ["SUCCEEDING", "SUCCEEDED"]:
                return
            time.sleep(1)  # Slow down requests to Flyte

    def get_first_node(self) -> NodeExecution:
        """Return the first node part of a workflow execution."""
        # Note: this functionality is not on current `FlyteRemote`
        while True:
            nodes = self.client.list_node_executions(
                self.workflow_execution.id,
                sort_by=Sort(key="started_at", direction="ASCENDING"),
            )[0]
            if len(nodes) > 1:
                # Wait until first node after "start-node"
                return nodes[0]
            time.sleep(1)  # Slow down requests to Flyte

Describe alternatives you've considered N/A Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2502 [BUG] [flytekit] Improve handling around StructuredDataset and other lossy types in Flyte remote Issue created by wild-endeavor Description If you

remote.fetch

a task or workflow or launch plan where one of the inputs is a StructuredDataset, and then try to execute it, flytekit will try to "guess" the interface for that structured dataset input and the type that it will come up with is the Python/flytekit

StructuredDataset

class. This is correct, but when we go and try to create the execution, we need to translate the dataframe from a pd.DataFrame or whatever instance into a StructuredDataset Literal. Since flytekit thinks the type annotation is a Python StructuredDataset, it will try to look it up in the list of formats/encoders it has and fail because it's not a real dataframe type. An example stack trace:

Traceback (most recent call last):
  File "/Users/nielsbantilan/miniconda3/envs/unionml/bin/unionml", line 33, in <module>
    sys.exit(load_entry_point('unionml', 'console_scripts', 'unionml')())
  File "/Users/nielsbantilan/miniconda3/envs/unionml/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/nielsbantilan/miniconda3/envs/unionml/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/nielsbantilan/miniconda3/envs/unionml/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/nielsbantilan/miniconda3/envs/unionml/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/nielsbantilan/miniconda3/envs/unionml/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/nielsbantilan/miniconda3/envs/unionml/lib/python3.9/site-packages/typer/main.py", line 500, in wrapper
    return callback(**use_params)  # type: ignore
  File "/Users/nielsbantilan/git/unionml/unionml/cli.py", line 99, in predict
    predictions = model.remote_predict(app_version, model_version, wait=True, **prediction_inputs)
  File "/Users/nielsbantilan/git/unionml/unionml/model.py", line 535, in remote_predict
    execution = self._remote.execute(
  File "/Users/nielsbantilan/miniconda3/envs/unionml/lib/python3.9/site-packages/flytekit/remote/remote.py", line 796, in execute
    return self.execute_remote_wf(
  File "/Users/nielsbantilan/miniconda3/envs/unionml/lib/python3.9/site-packages/flytekit/remote/remote.py", line 889, in execute_remote_wf
    return self.execute_remote_task_lp(
  File "/Users/nielsbantilan/miniconda3/envs/unionml/lib/python3.9/site-packages/flytekit/remote/remote.py", line 862, in execute_remote_task_lp
    return self._execute(
  File "/Users/nielsbantilan/miniconda3/envs/unionml/lib/python3.9/site-packages/flytekit/remote/remote.py", line 658, in _execute
    lit = TypeEngine.to_literal(ctx, v, hint, variable.type)
  File "/Users/nielsbantilan/miniconda3/envs/unionml/lib/python3.9/site-packages/flytekit/core/type_engine.py", line 696, in to_literal
    lv = transformer.to_literal(ctx, python_val, python_type, expected)
  File "/Users/nielsbantilan/miniconda3/envs/unionml/lib/python3.9/site-packages/flytekit/types/structured/structured_dataset.py", line 486, in to_literal
    fmt = self.DEFAULT_FORMATS[python_type]
KeyError: <class 'flytekit.types.structured.structured_dataset.StructuredDataset'>

We need to improve the erroring/experience around this. Potential things include: • Asking the user to provide a StructuredDataset instance instead of the raw dataframe? • Continuing to error but informing the user to provide the type_hints map. Misc Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2501 [Docs] Allow selecting different stable versions for documentation Issue created by stef-stripe Docs always show

v: latest

which contains latest github code state. We are installing flyte through pypi, so we only have access to the latest stable push, which means we sometimes see incorrect docs. It's also unlikely we'll always be able to stay on the latest version, so being able to see fixed versions of the documentation would be incredibly useful. flyteorg/flyte

#2500 [Docs] Update OAuth Authorization Service section Issue created by AlekhyaSasi Description Update the below doc to clearly explain this discussion - Slack Archives Related page Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte