Hi! I have a single cluster deployment of Flyte. For authorization I am trying to setup OpenID with Azure AD. I have followed Authenticating in Flyte - Flyte, and the relevant part of my configuration currently looks like this:

server: 
      security:
        secure: false
        useAuth: true
auth:
      authorizedUris:
        - https://<my public domain>
        - <http://flyteadmin:80>
        - <http://flyteadmin.mlops-services.svc.cluster.local:80>
      userAuth:
        redirectUrl: https://<my app url>/callback # Also added to redirect URI:s in Azure portal
        openId:
          baseUrl: <https://login.microsoftonline.com/><my tenant>/v2.0
          scopes:
            - email
            - openid
          clientId: <My client id from Azure portal>

I have the client secret in flyte-admin-secrets:

data:
    oidc_client_secret: <base64 encoded client_secret from Azure portal>

Ingress rule looks like this:

- backend:
    service:
      name: flyteadmin
      port:
        number: 80
    path: /callback
    pathType: ImplementationSpecific
- backend:
    service:
      name: flyteadmin
      port: 
        number: 80
    path: /callback/*
    pathType: ImplementationSpecific

Trying to access the public domain, the response is:

Access to <my domain> was denied
You don't have the user rights to view this page
HTTP error 403

Looking at the logs from flyte admin it seems to be failing during the callback:

{"json":{},"level":"error","msg":"Failed to retrieve tokens from request, redirecting to login handler. Error: [EMPTY_OAUTH_TOKEN] Failure to retrieve cookie [flyte_idt], caused by: http: named cookie not present","ts":"2023-02-17T13:04:21Z"}

{"json":{},"level":"error","msg":"Error generating encrypted accesstoken cookie [SECURE_COOKIE_ERROR] Error creating secure cookie, caused by: securecookie: the value is too long","ts":"2023-02-17T13:04:22Z"}

{"json":{},"level":"error","msg":"Error setting encrypted JWT cookie [SECURE_COOKIE_ERROR] Error creating secure cookie, caused by: securecookie: the value is too long","ts":"2023-02-17T13:04:22Z"}

It seems the generated cookie only is of length ~1400 I have tried different changes in the flyteadmin configmap, with the same result. Any advice on how to continue here?

Does anyone know what happens in a multi-cluster scenario when one doesn’t configure Execution Cluster Labels? Can Flyte be configured to just load balance everything between the available clusters, without specifying teams? Or alternatively does anyone know whether there is a way to configure this with infrastructure as code/in the helm chart?

hello, does anyone know why i am seeing the

cache was disabled for this task

for a simple python task, when I have the cache enabled:

@task(cache=True, cache_version="1.0")
def simple_python_task(name: str):
    print(f"Hello {name}")

I feel like I'm missing something really stupid with the run command. I posted an image of the config. It is going to the right URL. The job gets spawned and will start to run, but the URL in the

pyflyte run

response is wrong. Could this be a bug?

Hi everyone, I'm trying to run a flyte workflow remotely using flyte run --remote but this is the error I encountered, it was fine when running locally. I ran into the same problem with the very first Getting Started module.

{"asctime": "2023-02-19 10:12:08,645", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:30081: Failed to connect to remote host: Connection refused\"\n\tdebug_error_string = \"UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:30081: Failed to connect to remote host: Connection refused {created_time:\"2023-02-19T10:12:08.645069-05:00\", grpc_status:14}\"\n>, sleeping 200ms and retrying"}
{"asctime": "2023-02-19 10:12:08,846", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:30081: Failed to connect to remote host: Connection refused\"\n\tdebug_error_string = \"UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:30081: Failed to connect to remote host: Connection refused {created_time:\"2023-02-19T10:12:08.846193-05:00\", grpc_status:14}\"\n>, sleeping 400ms and retrying"}
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/bin/pyflyte", line 8, in <module>
    sys.exit(main())
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/flytekit/clis/sdk_in_container/run.py", line 552, in _run
    remote_entity = remote.register_script(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/flytekit/remote/remote.py", line 800, in register_script
    upload_location, md5_bytes = fast_register_single_script(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/flytekit/tools/script_mode.py", line 111, in fast_register_single_script
    upload_location = create_upload_location_fn(content_md5=md5)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/flytekit/clients/friendly.py", line 998, in get_upload_signed_url
    return super(SynchronousFlyteClient, self).create_upload_location(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/flytekit/clients/raw.py", line 43, in handler
    return fn(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/flytekit/clients/raw.py", line 871, in create_upload_location
    return self._dataproxy_stub.CreateUploadLocation(create_upload_location_request, metadata=self._metadata)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:30081: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:30081: Failed to connect to remote host: Connection refused {grpc_status:14, created_time:"2023-02-19T10:12:09.247133-05:00"}"
>

Hi, I need to port forward a service inside the flyte sandbox container. I was able to do this by launching the container as where the service is running at 127.0.0.1:8501.

docker create --privileged -p 0.0.0.0:30082:30082 -p 0.0.0.0:30084:30084 -p 0.0.0.0:30086:30086 -p 0.0.0.0:30088:30088 -p 0.0.0.0:30089:30089 -p 0.0.0.0:30080:30080 -p 0.0.0.0:30081:30081 -p 127.0.0.1:8501:8501 --env SANDBOX=1 --env KUBERNETES_API_PORT=30086 --env FLYTE_HOST=localhost:30081 --env FLYTE_AWS_ENDPOINT=<http://localhost:30084> --mount type=bind,source=/home/aswanth/.flyte,target=/etc/rancher/ --mount type=bind,source=/home/aswanth/.flyte/state,target=/srv/flyte --name flyte-sandbox <http://cr.flyte.org/flyteorg/flyte-sandbox:dind-f69fb09ca189e8bf57e1a6a12db168274f640d15|cr.flyte.org/flyteorg/flyte-sandbox:dind-f69fb09ca189e8bf57e1a6a12db168274f640d15>

after this when i try to register my workflows using pyflyte register i am gettting the following error

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:

status = StatusCode.UNAVAILABLE

details = "failed to connect to all addresses; last error: UNAVAILABLE: Socket closed"

debug_error_string = "UNKNOWN:Failed to pick subchannel {created_time:"2023-02-20T12:41:31.69117013+05:30", children:[UNKNOWN:failed to connect to all addresses; last error: UNAVAILABLE: Socket closed {created_time:"2023-02-20T12:41:31.691159352+05:30", grpc_status:14}]}"

how can i achieve my intention? Thanks in advance

S3 Storage: Separate Backends for Metadata and for User Data? Cheers everyone, We have the following problem: Our Workflow & Task authors want to read & write files stored in a S3 backend, ideally via the Flyte Files. However, we don't want to have Flyte Metadata in the same endpoint. Can I configure the Flyte Pods executing the Tasks to have access to the "user S3 backend", without Flyte trying to store all its metadata there as well? Details: Locally on their laptops, we configured the

~/.flyte/config.yaml

, providing user-backend values for

storage.connection.endpoint, access-key, secret-key

etc, and executing the workflow via

python myworkflow.py

works like a charm -

FlyteFile("<s3://my-bucket/my-file.csv>").download()

accesses the user-backend configured in the

config.yaml

and downloads the file. Now regarding the Flyte deployment, I'm a bit lost which component (Admin, Propeller, ...) to configure so that the Pods have access to the s3-user-backend, without automatically trying to write their metadata (e.g.,

<s3://flyte/metadata/propeller/myproject-development-ffba3b040e85d4801a9c/n1/data/0>

)?

flyte on GKE 1.25 Does someone run flyte already on a GKE 1.25 version while using workload identity? We noticed that it breaks fast registration of workflows as signed URLs can't be created:

...
  File "/opt/pyenv-root/versions/3.9.12/lib/python3.9/site-packages/flytekit/remote/remote.py", line 580, in fast_package
    return self._upload_file(pathlib.Path(zip_file))
  File "/opt/pyenv-root/versions/3.9.12/lib/python3.9/site-packages/flytekit/remote/remote.py", line 598, in _upload_file
    upload_location = self.client.get_upload_signed_url(
  File "/opt/pyenv-root/versions/3.9.12/lib/python3.9/site-packages/flytekit/clients/friendly.py", line 998, in get_upload_signed_url
    return super(SynchronousFlyteClient, self).create_upload_location(
  File "/opt/pyenv-root/versions/3.9.12/lib/python3.9/site-packages/flytekit/clients/raw.py", line 41, in handler
    return fn(*args, **kwargs)
  File "/opt/pyenv-root/versions/3.9.12/lib/python3.9/site-packages/flytekit/clients/raw.py", line 856, in create_upload_location
    return self._dataproxy_stub.CreateUploadLocation(create_upload_location_request, metadata=self._metadata)
  File "/opt/pyenv-root/versions/3.9.12/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/opt/pyenv-root/versions/3.9.12/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.INTERNAL
        details = "failed to create a signed url. Error: unable to sign bytes: googleapi: Error 403: Request had insufficient authentication scopes.
Details:
[
  {
    "@type": "<http://type.googleapis.com/google.rpc.ErrorInfo|type.googleapis.com/google.rpc.ErrorInfo>",
    "domain": "<http://googleapis.com|googleapis.com>",
    "metadata": {
      "method": "google.iam.credentials.v1.IAMCredentials.SignBlob",
      "service": "<http://iamcredentials.googleapis.com|iamcredentials.googleapis.com>"
    },
    "reason": "ACCESS_TOKEN_SCOPE_INSUFFICIENT"
  }
]"
        debug_error_string = "UNKNOWN:Error received from peer ipv4:{removed} {created_time:"2023-02-20T11:01:31.243469439+00:00", grpc_status:13, grpc_message:"failed to create a signed url. Error: unable to sign bytes: googleapi: Error 403: Request had insufficient authentication scopes.\nDetails:\n[\n  {\n    \"@type\": \"<http://type.googleapis.com/google.rpc.ErrorInfo\|type.googleapis.com/google.rpc.ErrorInfo\>",\n    \"domain\": \"<http://googleapis.com|googleapis.com>\",\n    \"metadata\": {\n      \"method\": \"google.iam.credentials.v1.IAMCredentials.SignBlob\",\n      \"service\": \"<http://iamcredentials.googleapis.com|iamcredentials.googleapis.com>\"\n    },\n    \"reason\": \"ACCESS_TOKEN_SCOPE_INSUFFICIENT\"\n  }\n]"}"

We noticed this issue only appears on our clusters running GKE version

1.25.5

, it does not appear on the clusters still running

1.24.9

Is it a bug that when workflow is archived none of the launch plans are?

Hi everyone… getting a ‘NoneType’ object has no attribute ‘inputs’ trying to execute a launch plan. the workflow doesn’t have inputs parameters. any ideas?

from flytekit import Email, LaunchPlan
from flytekit.models.core.execution import WorkflowExecutionPhase
from flytekit.remote import FlyteRemote
from flytekit.configuration import Config

remote = FlyteRemote(
    config=Config.for_endpoint(endpoint="dns:///flyte.eks.tech"),
    default_project="flytesnacks",
    default_domain="development",
)

wf = remote.fetch_workflow(name="parent_workflows.workflows.example.my_wf", version="Mh7JCmzuS6MJr2XWw5ST-Q==", project="flytesnacks", domain="development")
print(wf) #prints correctly
lp = LaunchPlan.get_or_create(wf)
print(lp) #prints correctly
execution = remote.execute(lp, inputs={}, execution_name="test lanuch plan", wait=False) #fails here 'NoneType' object has no attribute 'inputs'
print(execution)

Hey guys, Trying to set up auth with the flyte-binary and having some issues with the secrets. What I have so far in my values:

auth:
    enabled: true
    oidc:
      baseUrl: "<azureAD oidc application>"
      clientId: "<oidc id>"
      clientSecret: <base64 encoded oicd secret>
    internal:
      clientSecret: <base64 encoded secret>
      clientSecretHash: "<bcrypt hash of above>"

I guess that may not be terribly informative? Point being, I'm quite confident that the

oidc.baseUrl

and

oidc.clientID

are correct. However, when I install the chart, the

gen-admin-auth-secret

container errors out with:

* error decoding 'appAuth.selfAuthServer.staticClients[flytepropeller].client_secret': illegal base64 data at input byte 0

Couple of questions: • Which client secret is this? • I've tried base64 encoding locally with python and here • Is it possible to store these values in a k8s secret? • Is this really all the auth setup needed? I'm migrating from a pre-binary setup to the binary and there's a lot of complexity in the former chart which is not present in the latter Any pointers would be much appreciated ☺️

Hi, I'm running Flyte on a single cluster deployment (GCP), using helm (https://flyteorg.github.io/flyte v1.3.0) and trying to set up auth using OpenId Connect. I've followed the instructions here: https://docs.flyte.org/en/v1.0.0/deployment/cluster_config/auth_setup.html#openid-connect and managed to set up user authentication with browser and flytectl (i.e. I'm able to log in and list projects). However I'm unable to start workflows; I see errors in

flytescheduler

and

flytepropeller

logs. Logs from

flytescheduler

:

flytescheduler-check panic: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: oauth2: cannot fetch token: 401 Unauthorized                                           
flytescheduler-check Response: {"error":"invalid_client","error_description":"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method)."}                                                                                                                                                                                                        │
flytescheduler-check                                                                                                                                                                                      
flytescheduler-check goroutine 1 [running]:                                                                                                                                                               
flytescheduler-check main.main()                                                                                                                                                                          
flytescheduler-check     /go/src/github.com/flyteorg/flyteadmin/cmd/scheduler/main.go:12 +0x85

My current setup is:

configmap:
  adminServer:
    auth:
      authorizedUris:
        - <internal domain>
        - <https://localhost:30081>
        - <http://flyteadmin:80>
        - <http://flyteadmin.flyte.svc.cluster.local:80>
      userAuth:
        openId:
          baseUrl: <https://accounts.google.com>
          clientId: <id>.<http://apps.googleusercontent.com|apps.googleusercontent.com>
          scopes:
            - profile
            - openid
    server:
      httpPort: 8088
      grpcPort: 8089
      security:
        secure: false
        useAuth: true
        allowCors: true
        allowedOrigins:
          # Accepting all domains for Sandbox installation
          - "*"
        allowedHeaders:
          - "Content-Type"

I've tried adding

appAuth

(same level as

userAuth

) but can't seem to figure out a proper configuration. Any help on what to do next would be much appreciated.

Are there any recommendations/best practices about version numbers in

cache_version="1.0"

in the scenario that two people are working in parallel on a workflow so that one doesn’t accidentally use cache results from the other engineer? Will it cause any problems if we even put the engineer’s name in the cache version?

I am trying to follow the single cluster deployment tutorial using EKS, S3 and RDS. My cluster is getting bottlenecked with some problems appear to be caused by the Database. I was just wondering if I have configured my eks-start.yaml file correctly?

Hi everyone, I have a streamlit UI dashboard running inside the flyte sandbox as a kubernetes service. is there a way to monitor and trigger workflows from this dashboard?

Hi team, is there a way to configure the s3 endpoint url. We’re looking to use interface endpoint to access s3. An example endpoint would look like: vpce-aaaabbbbcccc-dddd.vpce-svc-12345678.ap-east-1.vpce.amazonaws.com https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html#vpce-private-dns:~:text=The%20D[…]ts%20are%20publicly%20resolvable

Hello guys I've set up Kubernetes and Flyte on AWS and then port forwarded to my local machine. However, when I try to register a workflow, I get this error:

error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER

I have a question about the protobuf versions required by flyteidl:

onnx

and

tensorboardX

for instance don’t support protobuf 4 yet (see here and here) which blocks us from using newer versions of flyteidl and as a consequence flytekit.

tensorflow

also only supported 4.21 as of three weeks ago but this isn’t even part of the newest release yet so I would have to install from this commit. I understand this is annoying from the flyte side that these libraries move slower but is there any small chance 3.x could be supported again a little while longer? 🙈

Is it possible to cancel/terminate individual tasks in a workflow somehow?

Hello there, below is a picture of my launch plan versions, the older one runs every five minutes, the new one runs every minutes (cron expression). Just to confirm the new version will overtake the old one correct, meaning the every five minute one will now become every minute? Can someone please confirm/deny my assumption?

Screen Shot 2023-02-15 at 6.26.12 PM.png

Hello people, as

pyflyte run --remote

"cannot import tasks and workflows from other files currently" I tried going the way of "use

pyflyte register

to handle those". (quotations taking from the --help output of pyflyte run) With my current setup:

workflow/
|- ClassTest.py
|- example.py

And example.py using ClassTest.py like the following:

from ClassTest import TestClass
from flytekit import task, workflow

@workflow
def wf():
    test_class = TestClass()
    test_class.do_stuff()

I get

ModuleNotFoundError: No module named 'ClassTest'

. What am I missing?

Am I correct in understanding that Flyte does not support custom encoders and decoders for dataclasses_json.

class MyDataClass:
     foo: Foo = field(metadata=dict(encoder=encoder, decoder=decoder))

Is the only alternative to define and register a custom

TypeTransformer

? What's the logic behind requiring something comparatively cumbersome?

Hey everyone, I'm trying to set up auth with the flyte binary and while I've had some success (with help from you guys 🙂), I'm still running into a few issues. In particular, I'm using Azure as the OIDC provider and have everything working in the below snippet. This is one Azure application and the

oidc

section here allows me to access the UI via an ingress (defined via the chart) 🦜

auth:
    enabled: true
    oidc:
      baseUrl: <https://signin.hosted.unionai.cloud/oauth2/default>
      clientId: <IDP_CLIENT_ID>
      clientSecret: <IDP_CLIENT_SECRET>
    internal:
      clientSecret: <CC_PASSWD>
      clientSecretHash: <HASHED_CC_PASSWD>

I also have another azure application which, in the pre-flyte-binary setup that I am migrating from, was used to populate the values here:

configmap:
  adminServer:
   auth:
     ...
     appAuth:
       ...
       thirdPartyConfig:
         flyteClient:
           clientId: XXX
           redirectUri: <http://localhost:53593/callback>
           scopes:
             - XXX

In the flyte-binary chart, editing these values isn't an option but if you check out the

flyte-backend-flyte-binary-config

you can see some hardcoded values:

thirdPartyConfig:
  flyteClient:
    clientId: flytectl
    redirectUri: <http://localhost:53593/callback>
    scopes:
    - offline
    - all

Is it possible to overwrite these defaults? I tried the following in my flyte-binary chart values:

configuration:
  inline:
    auth:
      appAuth:
        thirdPartyConfig:
          flyteClient:
            clientId: XXX
            redirectUri: <http://localhost:53593/callback>
            scopes:
              - XXX

But I don't think this worked. Sorry for the long post, but any idea how I can override hard-coded

thirdPartyConfig

values in the helm chart? Thanks a lot for all the continued help 🙂

sorry to spam here - but please join the community sync - live now on Zoom - https://us06web.zoom.us/j/93875115830?pwd=YWZWOHl1ODRRVjhjVkxSV0pmZkJaZz09 This helps the community to know whos involved and if they are having problems and also share notes with each other

Could someone please point me to the default container image used for python tasks, can't seem to find it in the docs or in flytekit....

I have two questions: 1. How can I monitor the std out and std err while the task is running ? 2. Does

pyflyte register

not use the docker image when its packaging the workflow. We have some custom dependencies for the workflow to execute that aren’t present in the development environment.

Anyone know how to fix the issue when flyte sandbox doesn’t start. I’m just getting:

Waiting for cluster to come up...

Over and over again..

Hello guys. Is there some other option to pass --outputLocationPrefix? I know that this can be used for "flytectl register file ..." and can be set in web UI for workflows registered by pyflyte register. But maybe it can be passed somehow in code? Maybe as some environment variable etc?

Hello guys I'm having a new error. I packaged a project to Docker hub. And then used FlyteRemote to register the workflow to a remote Flyte cluster on AWD EKS -- adding the IMAGE_STR. However, when I fetch and execute the workflow, it raises a

ModuleNotFoundError

. What could I be doing wrong please?