TLDR; 403 forbidden errors using flytectl (oidc and oauth2 enabled with Okta) Hi everyone! I’m struggling to configure

flytectl

I deployed Flyte with Helm on GKE cluster using OIDC and OAuth2 with Okta. The ingress is

flyte.my.domain

exposed by GKE ingress controlle. The console works perfectly. This is how the

config.yaml

looks like:

admin:
  endpoint: dns:///flyte.my.domain
  authType: Pkce
  insecure: false

I keep getting this error:

PermissionDenied desc = unexpected HTTP status code received from server: 403 (Forbidden); malformed header: missing HTTP content-type

Am I missing something? Would appreciate your help! Thanks

Hi, new to Flyte and was wondering how one could change the "output" and "input" decks. Thanks.

Hi Community, whenever I try to launch the workflow, I get an error "OOMKilled". How to solve this? I am using demo sandbox (union sandbox). Earlier, this issue was not there. Since, I migrated from one server to another, I start facing these issues.

Is there a way to specify the output path for the two implicit outputs for NotebookTask (Jupyter)? Essentially the desired workflow is to load in the source Jupyter notebook from EFS and then output the rendered HTML and output notebook to some sort of scratch/ephemeral storage so that the output file paths don't overlap during execution of parallel tasks. Been running into some errors currently as the current papermill plugin appears to derive the output paths from the source file name and then appending "out.ipynb" and "out.html". It would be great if we could append a unique run ID to each outputted file and/or if we can manually the output path.

I follow the docs to deploy the flyte. But after I run the hello_world.py in remote, I got “Original exception: Unable to locate credentials” error. I’ve tried to search docs and issues, but found nothing about it.

Is there a recommended way to use secret in a ShellTask ?

I'm having issues accessing the grpc endpoint behind an AWS ELB. We have flyte installed by following this: https://docs.flyte.org/en/latest/deployment/deployment/cloud_production.html And we have the http endpoint exposed via alb w/o issue. We use the aws ingress controller to create the ALBs, so I've modified the grpc ingress definition to add the following annotations: alb.ingress.kubernetes.io/backend-protocol: HTTP alb.ingress.kubernetes.io/backend-protocol-version: GRPC alb.ingress.kubernetes.io/certificate-arn: <redacted> alb.ingress.kubernetes.io/healthcheck-path: /grpc.health.v1.Health alb.ingress.kubernetes.io/healthcheck-protocol: HTTP alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]' alb.ingress.kubernetes.io/scheme: internal alb.ingress.kubernetes.io/security-groups: EKS-ALB-sg alb.ingress.kubernetes.io/target-type: ip And the spec with: ingressClassName: alb rules: - host: <http://flyte-web-east2-grpc.api-platform-dev.aws-nonprod.fmrcloud.com|<>hostname> But whenever I try to use grpcurl to access the endpoint to list/describe I always get: $ grpcurl <hostname>:443 list Failed to list services: server does not support the reflection API $ grpcurl <hostname>:443 describe /flyteidl.service.IdentityService Failed to resolve symbol "/flyteidl.service.IdentityService": server does not support the reflection API When I try to use grpc_cli ls <hostname>:443 I get an error: Trying to connect an http1.x server I can describe the grpc endpoints when I port-forward the grpc service, so the server is working as expected. Has anyone else tried to do this?

Reminder: 🔴 Community meeting is starting now - Info: https://www.addevent.com/event/EA7823958

Hey i have a question regarding the max_parallelism option for running a workflow remotely. Even tough i configured max parellism it seems to always spin up 20 pods at a time. Anyone know whats going wrong here?

Why does the doc string not show up when the Union type is set to string? Is this a bug?

I'm trying to setup my dev environment. I installed flytekit and the requirements for flytesnacks in an venv as recommended here: https://docs.flyte.org/projects/cookbook/en/latest/userguide_setup.html However, my pyflyte looks broken. I can't run any commands as they all fail with: ModuleNotFoundError: No module named 'flyteidl.service.external_plugin_service_pb2_grpc' That module definitely isn't installed in my venv, and I don't see the proto for it in the flyteidl repo either. Have I installed a wrong version of flytekit somehow?

Hey can I get the execution id hash from inside a task? This is the thing from the UI that I want f4786acee61844e39b46

Hello, I am trying to use jinjasql in flyte task. jinjasql will need to read a templated sql file (a text file with extension .sql) from a subdir somewhere in the project. I can image the flyte task may fail due to path / file not found, and that is because the sql file is not being copied to the remote flyte server when pyflyte run or pyflyte register is executed. Am I correct? What will be the solution if I still want to use jinjasql? I am using flytekit 1.2.4.

I am noticing that the error shown in Flyte is not the error that I want to see. Maybe this is only an issue with Flyte on GCP, but it seems good change to actually show the actual error that I care about.

What format should the

caCertFilePath

be in for the .flyte/config? I've tried many different formats, but only .pem works for flytectl. Nothing works for pyflyte though. pyflyte thows this error when I use a .pem: Failed with Unknown Exception <class 'TypeError'> Reason: expected certificate to be bytes, got <class 'OpenSSL.crypto.X509'> expected certificate to be bytes, got <class 'OpenSSL.crypto.X509'>

Hi, I am using the

flyte-binary

helm chart to setup external auth server. Unfortunately, not being successfully, the below config is not getting propagated to the configmap. Setting

enableAuthServer=false

is having no effect. Has anyone successfully used

flyte-binary

helm chart to setup an external server?

configuration:
  auth:
    enabled: true
    enableAuthServer: false
    oidc:
      baseUrl: "okta-server"
      clientId: "my_id"
      clientSecret: "my_secret-"
    internal:
      # propeller
      clientId: "my_id"
      clientSecret: "my_secret-"
      clientSecretHash: "hash
    flyteClient:
      clientId: my_id"
      redirectUri: <http://localhost:53593/callback>
      scopes:
        - all
        - offline
    authorizedUris:
      - https://${app_name}.${env}.<http://cloud.com|cloud.com>
      - <http://flyteadmin:80>
      - <http://flyteadmin.flyte.svc.cluster.local:80>

I am running into an issue with the flytesnacks bigquery example that I can't figure out.

[3/3] currentAttempt done. Last Error: SYSTEM::Traceback (most recent call last):

      File "/usr/local/lib/python3.10/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
        return wrapped(*args, **kwargs)
      File "/usr/local/lib/python3.10/site-packages/flytekit/core/base_task.py", line 530, in dispatch_execute
        raise type(exc)(msg) from exc

Message:

    Failed to convert inputs of task 'flytesnacks.convert_bq_table_to_pandas_dataframe':
  Protocol not known: bq

SYSTEM ERROR! Contact platform administrators.

Hi! I have kind of an open-ended ask, but I was wondering if anyone has experience using Flyte in production with purpose-built batch schedulers like Apache Yunikorn or Volcano? If so, I'd love to chat to understand more about your experience and any pragmatic details for what the integration process looks like. Bonus points if you're also using any Flyte plugins that rely on operators to create ephemeral clusters -- e.g., Spark or Ray -- and might have some insights to share about how those interact w/ a batch scheduler.

hi! i was wondering how is initContainers set in task details if i didn’t specify it anywhere? i find that the container’s stringValue is usually the correct container i set in container_image, but the initContainer’s stringValue is often the container image i used in previous registration.

Hello, I need to send slack notification upon workflow failure, aborted, etc. by the Flyte framework. The notifications provided by https://docs.flyte.org/projects/cookbook/en/latest/auto/deployment/lp_notifications.html#sphx-glr-auto-deployment-lp-notifications-py would have worked for me. I need to have a Slack email ID generated through Slack email integration. However, our company policy doesn’t allow Slack email integration. The allowed slack notification is through Incoming webhook. It provides me with a webhook URL. Does anyone know if Flyte can use webhook URL to automatically send a slack notification when the flyte workflow / task failed or aborted?

Hey team! One question here. Flytekit latest versions use fsspec under the hood for interacting with remote storages AFAIK. Do we need to manually install the underlying driver for this to work? For example,

s3fs

or

gcsfs

, or should it work transparently? One of our projects is having issues due to this, and I’m not sure if it’s a problem on the project’s dependencies, or something that we need to specifically install besides flytekit. Thanks!

Hello, I am a beginner at using Flyte. I created a workflow where the final task is to save an object in my own s3 bucket based on Flyte demo sandbox cluster. I tried to pass in the environment variables through Dockerfile as follows:

ENV AWS_ACCESS_KEY_ID [some value]

ENV AWS_SECRET_ACCESS_KEY [some value]

ENV AWS_SESSION_TOKEN [some value]

But I keep getting an error message. Is there a way around this? Or do i need to spin up my own cluster?

I'm trying to work around the issue in this thread: https://flyte-org.slack.com/archives/CP2HDHKE1/p1685547623758749 I added

insecureSkipVerify: true

to my config.yaml, and when I try to do a

pyflyte run -remote

I get this error: Failed with Exception Code: SYSTEM:Unknown RPC Failed, with Status: StatusCode.UNAVAILABLE details: unavailable Debug string UNKNOWN:Error received from peer {created_time:"2023-06-01T105201.368208-04:00", grpc_status:14, grpc_message:"unavailable"} We are accessing through an AWS ALB, and

flytectl get project

works so I think the grpc routing is working. I'm guessing the grpc error message is because python's grpc doesn't like invalid ssl? If I'm correct, does the error message I am receiving seem inline with expectations if the ssl cert isn't verified?

Who would be the best person to talk with about building a new data processing integration? (ie https://flyte.org/integrations)

Hello Flyte folks, so I am trying to use Optional type in our workflow, however i have been seeing this error, i have reproduced in a sandbox as well. More 🧵

Hello, I'm new to Flyte, but I was wondering how I might go about customizing the flyte console. I am considering adding a button to it. The reason would be to add additional links to a logs sink when errors surface. I do understand that there is a specific way to configure logging links via the Helm chart, but I was wondering if I'd be able to modify the UI on my end to achieve that functionality instead. Is that possible, and if so, would someone be able to point me in the right direction with regard to that?

Hello, How can I tell what minimum version of flytekit is that supports ImageSpec?

Hi all, another side discussion from the this thread https://flyte-org.slack.com/archives/CP2HDHKE1/p1685654534312309 Does flyte classify dynamic workflow compilation error as system error? because flytepropeller kept retrying it until it exhausts system retry limit

Hi Flyte team, How can I add labels to Ray Head and Worker nodes? Thanks