Hi everyone! I just created this channel to exchange knowledge and learnings running Flyte on GCP. Feel free to add anyone you know is also a GCP user @Ariel Kaspit is facing issues in the following situation: 1. Deployed `flyte-core`chart succesfully 2. Requires to enable developers to access Flyte through flytectl with PKCE but when they try, they get

403

errors (described here) 3. They're using the native GKE ingress controller, including the

<http://cloud.google.com/app-protocols|cloud.google.com/app-protocols>: '{"grpc":"HTTP2"}'

annotation. Not sure if that's enough for Ingress to route gRPC traffic @Fabio Grätz I remember you're using Flyte on GCP, so I was wondering if you had ideas/recommendations from your experience Thanks!

Hey, yes, we run Flyte on GCP, happy to share what we do.

The GKE ingress controller is really nice because it supports identity aware proxy, basically google managed login screen that is placed before Flyte. We are required to use this for security reasons.

The GKE ingress, however, cannot handle the gRPC readiness and liveness probes of flyteadmin so configuring flyteadmin as a backend to the GKE ingress will not work.

You can deploy the nginx ingress in GKE and use it in combination with cert manager. The Flyte GCP docs were updated with what was msising by I think you @David Espejo (he/him) after I tried this (thanks btw). If following this setup, pkce will work. Happy to help debug in case it doesn’t

If you decide to use the GCP Ingress (which I would only do if you want to use google identity aware proxy), you will need to have some kind of additional reverse proxy behind the GCP ingress which implements the routing to the flyte backens (because of the readiness probe issue mentioned above). You could use an nginx for this. We use the istio service mesh.

HOWEVER: we run flyte in unauthenticated mode behind identity aware proxy which is not super nice. We gave flyteadmin an additional internal load balancer service which allows us to access it with flytectl, pyflyte, … from within the VPC. In our setup engineers cannot register workflows with pyflyte through the GKE ingress/identity aware proxy. It’s on my todo list to try to get this to work but until now it hasn’t been a priority.

What is your reason for going with the GKE ingress over the nginx ingress @Ariel Kaspit?

Unless you want/have to use identity aware proxy, I think the nginx ingress is the much simpler path.

Wow! That sounds great, I didn’t know actually GKE ingress behaves that way. I will definitely go to try and deploy nginx ingress controller

So first, @Fabio Grätz thank you for your help! Second, we use GKE ingress for all our deployments so we didn’t see any reason to use other ingress controller

@David Espejo (he/him) this reminds me that maybe in this section of the GCP setup we should remove the mentioning of the GCP ManagedCertificate resource and of the GKE ingress? I’m pretty sure that out of the box this will not work because of the issues outlined above.

@Ariel Kaspit if you use the GKE ingress, I guess you configured the routing rules for flyteadmin and flyteconsole there. Is the flytedamin backend shown as healthy for you in the GKE ingress?

At least half a year ago if I’m not mistaken this wouldn’t have worked. (Because of limitations of the GKE ingress, not flyte)

https://github.com/flyteorg/flyte/issues/3730 @David Espejo (he/him)

@Fabio Grätz Hi Fabio, how’s it going? I have few questions regarding your workaround… 1. In your configuration, did you enable the

separateGrpcIngress

? I understand from the documentation that it is Required for certain ingress controllers like nginx. 2. After deploying nginx (which uses our own tls certificate, not self-signed with cert-manager), I’m getting 502 bad gateway errors accessing the console (flyte.my.domain)… In the

flyteadmin

logs I see this authentication error:

Failed to refresh tokens. Restarting login flow. Error: [TOKEN_REFRESH_FAILURE] Error refreshing token, caused by: oauth2: cannot fetch token: 400 Bad Request

(it worth mentioning I don’t see any errors in Okta, which we configured there the authorization server) 3. Regarding accessing within the CLI, while trying to use

flytectl

I’m still getting the same authentication error:

PermissionDenied desc = unexpected HTTP status code received from server: 403 (Forbidden); malformed header: missing HTTP content-type

- Did you experience this thing? BTW, I don’t understand from your answer if you’re using

flytectl/pyflyte

? Thanks in advanced! And sorry for all the questions, I appreciate your help!

Hey everyone! I have a question regarding RBAC in Flyte (I know it’s not supported). How do you manage permissions for different domains (development, staging, production)?

Hi All , working on setting up Flyte on GKE as part of a side project. Does anyone have a document on how to deploy the helm chart on GKE cluster using helm_release resource block of terraform? I'm trying to make sure all the changes are tracked via terraform for me to easily pull apart the infra after use.

Hi all, we have a flyte-core deployment running on GKE since this winter, version 1.2.0. Been trying to upgrade us towards 1.8.0, but the helm_release terraform block isn’t completing due to flyteconsole’s health check. We have load-balancers with ingress class “gce-internal” at the moment, and I think it is the load balancer’s health check that is failing. Has anyone had any similar problems upgrading in the past six months?

Hi everyone! I'm also in a situation where I need to deploy Flyte to GCP based on the official helm chart. Almost everything is up and running, but I'm stuck with a

flytectl

problem which I can't solve for now; maybe someone will be able to help me with it?

$ HTTPS_PROXY=localhost:8080 flytectl --admin.insecureSkipVerify -d development -p flyteexamples get tasks 
Error: Connection Info: [Endpoint: dns:///flyte.dexterenergyservices.com, InsecureConnection?: false, AuthMode: Pkce]: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); transport: received unexpected content-type "text/html; charset=UTF-8"

I'm using mitmproxy to intercept the

flytectl

traffic (hence the

HTTPS_PROXY

env var and

--admin.insecureSkipVerify

additions) and I can see that

flytectl

issues a request to

/flyteidl.service.AdminService/ListTasks

endpoint. It results in HTTP 502 response from the GCP Load balancer claiming that "server encountered a temporary error". This is where I am stuck for now - I do have

flyteadmin

Service running and the corresponding

flyteadmin

Pods also green and pass the healthchecks, but I can't figure out how to confirm if

flyteadmin

is actually running and responding. Best thing I've found so far is that if I request the

/me

endpoint (which is sent to the HTTP interface of the

flyteadmin:80

service), it responds with the following response:

{
  "error": "unknown service flyteidl.service.IdentityService",
  "code": 12,
  "message": "unknown service flyteidl.service.IdentityService"
}

And that looks suspicious :) Maybe it's a well-known problem? I would appreciate any help with it

Hi All , I was able to get the basic setup for flyte-binary running on GCP (sort of) and the init DB container that it spins up succeeded , however it still fails while trying to connect to Cloud SQL instance. I have allowed the subnet that contains the cluster's external endpoint at Cloud SQL end. The PG instance thats being used PG 14.7 and I overwrote the values file to use the 14-alpine image. Is v1.8.0 flyte-binary helmchart supposed to be compatible with PG 15? The services are not in two different subnets and we're using the default VPC that is present. Below is the error on logs while checking the crashloopbackoff issue: [error] failed to initialize database, got error failed to connect to `host=XX.XXX.XXX.XXX user=postgres database=flyte-db-backend`: server error (FATAL: database "flyte-db-backend" does not exist (SQLSTATE 3D000)) The host= is referencing to the DB IP which is set using the following: set { name = "configuration.database.host" value = google_sql_database_instance.flyte_db_backend.public_ip_address } Image tag was overwritten using: set { name = "deployment.waitForDB.image.tag" value = "14-alpine" } Values reference taken for flyte-binary: https://github.com/flyteorg/flyte/blob/master/charts/flyte-binary/values.yaml

I’m also trying to get flyte setup on GCP. My company has a strict requirement to use IAP everywhere. We’ve been able to get authentication to work on the console but I can’t get access via the flytectl to work. Any tips here? recent thread https://flyte-org.slack.com/archives/CP2HDHKE1/p1690818970921269

Does anyone configure notifications for workflows on GKE? It seems like it is supported for aws only? https://docs.flyte.org/en/latest/deployment/configuration/notifications.html#config

Did anyone run into issue with Flyte retrieving items from storage blob?

[1/1] currentAttempt done. Last Error: USER::load_distribution(additional_distribution, dest_dir)      │
│                                                                              │
│ /usr/local/lib/python3.10/site-packages/flytekit/core/utils.py:295 in        │
│ wrapper                                                                      │
│                                                                              │
│ ❱ 295 │   │   │   │   return func(*args, **kwargs)                           │
│                                                                              │
│ /usr/local/lib/python3.10/site-packages/flytekit/tools/fast_registration.py: │
│ 113 in download_distribution                                                 │
│                                                                              │
│ ❱ 113 │   FlyteContextManager.current_context().file_access.get_data(additio │
│                                                                              │
│ /usr/local/lib/python3.10/site-packages/flytekit/core/data_persistence.py:30 │
│ 1 in get_data                                                                │
│                                                                              │
│ ❱ 301 │   │   │   raise FlyteAssertion(                                      │
╰──────────────────────────────────────────────────────────────────────────────╯
FlyteAssertion: Failed to get data from 
<gs://flyte-blob-storage/flytesnacks/development/MHEQCB3WRLH247GOTRY3CZQIWE======>
/fastea66e2416c583fdc6995e9e030c414bf.tar.gz to /root/ (recursive=False).

Original exception: 403, message='Forbidden', 
url=URL('<https://storage.googleapis.com/download/storage/v1/b/flyte-blob-storage>
/o/flytesnacks%2Fdevelopment%2FMHEQCB3WRLH247GOTRY3CZQIWE======%2Ffastea66e2416c
583fdc6995e9e030c414bf.tar.gz?alt=media')

The

https

link seems to include extra entries which results in

403 error

I am working on deploying flyte on GCP using a managed HTTPS load balancer and HTTP/2 to communicate to the flyteadmin GRPC backend. I got most things working but ran into some issues getting GRPC healthcheck / requests to work. I came across this thread https://discuss.flyte.org/t/835/u01ubdc4e1l-yes-i-am-going-to-send-an-email-soon-to-gcp-ther and was wondering if anyone here has a successful approach to the issue

Hi all, Newbie here! we are considering using Flyte for LLM fine-tuning workflows. I am trying to deploy single cluster simple cloud deployment following this : https://docs.flyte.org/en/latest/deployment/deployment/cloud_simple.html - Created a Postgres (CloudSQL) instance, database, user - Created a GCS bucket "flyte-poc-data" - Created Service Account "dev01-flyte-poc-sa" with GKE,GCS Admin access. Since the documentation doesn't include a config for GKE , I created values.yml as follows:

configuration:
  database:
    username: flyte_user
    password: {password}
    host: 192.{db ip}
    port: 5432
    dbname: flyte
  storage:
    metadataContainer: flyte-poc-data
    userDataContainer: flyte-poc-data
    provider: gcs
    providerConfig:
      gcs:
        project: "{our GCP project id}"
serviceAccount:
  create: false
  name: dev01-flyte-poc-sa

helm install executed without any errors. But I only see Services and Deployments in k8s. There are NO pods created. Any idea ?

helm install flyte-backend flyteorg/flyte-binary --namespace flyte --values gcp-values.yml

W0911 15:41:23.101181   33738 warnings.go:70] autopilot-default-resources-mutator:Autopilot updated Deployment flyte/flyte-backend-flyte-binary: defaulted unspecified resources for containers [wait-for-db, flyte] (see <http://g.co/gke/autopilot-defaults>)
NAME: flyte-backend
LAST DEPLOYED: Mon Sep 11 15:41:20 2023
NAMESPACE: flyte
STATUS: deployed
REVISION: 1
TEST SUITE: None

#ask-the-community hi team, i managed to setup a simple cluster on GKE . Now I want to check how workflows spin off GPUs and modified the hello_world example as follows :

from flytekit import task, workflow, Resources


@task(requests=Resources(cpu="1", gpu="1", mem="1Gi"),
	  container_image="<http://ghcr.io/flyteorg/flytecookbook:kfpytorch-latest|ghcr.io/flyteorg/flytecookbook:kfpytorch-latest>",)
def say_hello() -> str:
    return "Hello, World!"

@workflow
def hello_world_wf_gpu() -> str:
    res = say_hello()
    return res

if __name__ == "__main__":
    print(f"Running my_wf() {hello_world_wf_gpu()}")

When I run this workflow, I get the follow error . Any idea how to fix this ?