hi i deployed flyte to my local k8s cluster i can view the w

Question

hi i deployed flyte to my local k8s cluster i can view the web ui but can not use the command line to execute workflows the following are is the error any way to workaround thank you

Kevin Su · Answer

did you set insecure in flytectl config to false

盛毅 · Answer

this is the config yaml

Kevin Su · Answer

try to add `insecure skip verify true` to admin config

盛毅 · Answer

< Kevin Su> hi kevin same error

盛毅 · Answer

< Kevin Su> seems like the command line used but actually it is

Kevin Su · Answer

how did you install flyte use helm install

盛毅 · Answer

helm install

Kevin Su · Answer

cc < David Espejo he him > did you run into this issue before

Kevin Su · Answer

wait could you set insecure to true instead

盛毅 · Answer

already set it true still got the same error

David Espejo (he/him) · Answer

hey 盛毅 what chart did you install

David Espejo (he/him) · Answer

have you tried not specifying the port in `endpoint`

盛毅 · Answer

< David Espejo he him > hello David i used the flyte core helm chart and wanna test it in multi cluster the error message showed rpc request to a http server how to config the endpoint to use http protocol here are my config yaml Can you show me your config file thank you `admin ` ` For GRPC endpoints you might want to use dns flyte myexample com` `endpoint dns flyte nginxbeta k8s` ` endpoint dns 10 24 73 16 32332` `authType Pkce` `insecure true` ` insecure skip verify true` `logger ` `show source true` `level 0`

David Espejo (he/him) · Answer

sure this is my config file ```admin For GRPC endpoints you might want to use dns flyte myexample com endpoint dns flyte the hard way uniondemo run authType Pkce insecure false insecureSkipVerify true logger show source true level 6```

David Espejo (he/him) · Answer

the `http1 x` error message typically refers to a gRPC connection using the wrong port What are you using for Ingress can you share a `describe` of that resource

盛毅 · Answer

the ingress nginx ` root betam1 ~ kubectl n ingress nginx get svc` `NAME TYPE CLUSTER IP EXTERNAL IP PORT S AGE` `ingress nginx controller LoadBalancer 10 96 0 160 10 24 73 232 80 30973 TCP 443 30953 TCP 28m` `ingress nginx controller admission ClusterIP 10 96 3 112 lt none gt 443 TCP 35m` the ingress in flyte namespace ` root betam1 ~ kubectl n ingress nginx get svc` `NAME TYPE CLUSTER IP EXTERNAL IP PORT S AGE` `ingress nginx controller LoadBalancer 10 96 0 160 10 24 73 232 80 30973 TCP 443 30953 TCP 28m` `ingress nginx controller admission ClusterIP 10 96 3 112 lt none gt 443 TCP 35m` the flyte core ingress ` root betam1 ~ kubectl n flyte describe ing flyte core` `Name flyte core` `Labels lt none gt ` `Namespace flyte` `Address 10 24 73 21` `Default backend default http backend 80 lt error endpoints default http backend not found gt ` `Rules ` `Host Path Backends` ` ` `flyte nginxbeta k8s` ` console flyteconsole 80 10 244 203 248 8080 ` ` console flyteconsole 80 10 244 203 248 8080 ` ` api flyteadmin 80 10 244 9 75 8088 ` ` api flyteadmin 80 10 244 9 75 8088 ` ` healthcheck flyteadmin 80 10 244 9 75 8088 ` ` v1 flyteadmin 80 10 244 9 75 8088 ` ` well known flyteadmin 80 10 244 9 75 8088 ` ` well known flyteadmin 80 10 244 9 75 8088 ` ` login flyteadmin 80 10 244 9 75 8088 ` ` login flyteadmin 80 10 244 9 75 8088 ` ` logout flyteadmin 80 10 244 9 75 8088 ` ` logout flyteadmin 80 10 244 9 75 8088 ` ` callback flyteadmin 80 10 244 9 75 8088 ` ` callback flyteadmin 80 10 244 9 75 8088 ` ` me flyteadmin 80 10 244 9 75 8088 ` ` config flyteadmin 80 10 244 9 75 8088 ` ` config flyteadmin 80 10 244 9 75 8088 ` ` oauth2 flyteadmin 80 10 244 9 75 8088 ` ` oauth2 flyteadmin 80 10 244 9 75 8088 ` `Annotations console` `Events ` `Type Reason Age From Message` ` ` `Normal Sync 15m x2 over 16m nginx ingress controller Scheduled for sync` the ingress flyte core grpc ` root betam1 ~ kubectl n flyte describe ing flyte core grpc` `Name flyte core grpc` `Labels lt none gt ` `Namespace flyte` `Address 10 24 73 21` `Default backend default http backend 80 lt error endpoints default http backend not found gt ` `Rules ` `Host Path Backends` ` ` `flyte nginxbeta k8s` ` flyteidl service AdminService flyteadmin 81 10 244 9 75 8089 ` ` flyteidl service AdminService flyteadmin 81 10 244 9 75 8089 ` ` flyteidl service DataProxyService flyteadmin 81 10 244 9 75 8089 ` ` flyteidl service DataProxyService flyteadmin 81 10 244 9 75 8089 ` ` flyteidl service AuthMetadataService flyteadmin 81 10 244 9 75 8089 ` ` flyteidl service AuthMetadataService flyteadmin 81 10 244 9 75 8089 ` ` flyteidl service IdentityService flyteadmin 81 10 244 9 75 8089 ` ` flyteidl service IdentityService flyteadmin 81 10 244 9 75 8089 ` ` grpc health v1 Health flyteadmin 81 10 244 9 75 8089 ` ` grpc health v1 Health flyteadmin 81 10 244 9 75 8089 ` ` flyteidl service SignalService flyteadmin 81 10 244 9 75 8089 ` ` flyteidl service SignalService flyteadmin 81 10 244 9 75 8089 ` `Annotations console` ` GRPC` `Events ` `Type Reason Age From Message` ` ` `Normal Sync 15m x2 over 16m nginx ingress controller Scheduled for sync` the nginx ingress has a loadbalance IP 10 24 73 232 but in the ing describe above it got the address `Address 10 24 73 21` i have no idea what happended but i can successfully visit the flyte console web ui using the ip 10 24 73 232

盛毅 · Answer

< Kevin Su> < David Espejo he him > got those errors after i enabled auth section in the helm chart before that it is ok

David Espejo (he/him) · Answer

I don t see this annotation in your ingress resource `<http nginx ingress kubernetes io ssl redirect|nginx ingress kubernetes io ssl redirect> true ` Aslo for auth can you share your Helm values file

盛毅 · Answer

the auth part from helm values file and i deployed on my local k8s cluster `auth ` `grpcAuthorizationHeader flyte authorization ` `authorizedUris ` ` This should point at your public http Uri ` ` ` ` This will be used by internal services in the same namespace as flyteadmin` ` ` ` This will be used by internal services in the same cluster but different namespaces` ` assuming that flyte is installed in flyte namespace` ` Controls app authentication config` `appAuth ` `authServerType External` `externalAuthServer ` `allowedAudience flytepropeller` `baseUrl populate this value only if different from userauth baseUrl` `metadataUrl well known openid configuration ` `thirdPartyConfig ` `flyteClient ` `audience flytepropeller` `clientId flytepropeller` `redirectUri ` `scopes ` ` all` ` offline` ` access token` ` Controls user authentication` `userAuth ` `redirectUrl add the flyte url` `cookieSetting ` `domain flyte nginxbeta k8s add here the sitename for flyte` `sameSitePolicy DefaultMode` `openId ` `baseUrl ` `scopes ` ` profile` ` openid` ` offline access` `clientId flytecore` the following is part of the configmap flyte admin base config ```auth appAuth authServerType External externalAuthServer allowedAudience flytepropeller baseUrl metadataUrl well known openid configuration thirdPartyConfig flyteClient audience flytepropeller clientId flytepropeller redirectUri scopes all offline access token authorizedUris grpcAuthorizationHeader flyte authorization userAuth cookieSetting domain flyte nginxbeta k8s sameSitePolicy DefaultMode openId baseUrl clientId flytecore scopes profile openid offline access redirectUrl flyteadmin eventVersion 2 metadataStoragePrefix metadata admin metricsScope flyte profilerPort 10254 roleNameKey testing host server grpcPort 8089 httpPort 8088 security allowCors true allowedHeaders Content Type flyte authorization allowedOrigins secure false useAuth true```

David Espejo (he/him) · Answer

in some cases if redirects are not working because there s some networking component in the middle preventing it specifying the port in config helps like `endpoint dns flyte nginxbeta k8s 443` Also for auth please check out this guide in review to make sure it matches your deployment <https flyte 3837 org readthedocs build en 3837 deployment configuration auth setup html>

盛毅 · Answer

< David Espejo he him > hi David sorry to interrupt you again this is what i got when using flytectl i can visit the web ui cannot connect using cli is it something wrong the nginx ingess ``` root gpu2 flyte flytectl get projects json src viper go 398 level debug msg Config section storage updated No update handler registered ts 2023 07 17T22 22 55 04 00 json src viper go 398 level debug msg Config section root updated No update handler registered ts 2023 07 17T22 22 55 04 00 json src viper go 400 level debug msg Config section admin updated Firing updated event ts 2023 07 17T22 22 55 04 00 json src client go 63 level info msg Initialized Admin client ts 2023 07 17T22 22 55 04 00 json src auth interceptor go 86 level debug msg Request failed due to rpc error code = Unavailable desc = connection closed before server preface received If it s an unauthenticated error we will attempt to establish an authenticated context ts 2023 07 17T22 22 55 04 00 Error Connection Info Endpoint dns flyte nginx k8s 31821 InsecureConnection true AuthMode Pkce rpc error code = Unavailable desc = connection closed before server preface received json src main go 13 level error msg Connection Info Endpoint dns flyte nginx k8s 31821 InsecureConnection true AuthMode Pkce rpc error code = Unavailable desc = connection closed before server preface received ts 2023 07 17T22 22 55 04 00 ``` this is the log from nginx ingress controller ```2023 07 18T10 28 37 619817703+08 00 10 244 159 0 18 Jul 2023 10 28 37 +0800 PRI HTTP 2 0 400 150 0 0 001 9565a5c1e0480bab6e57defcec893209 2023 07 18T10 28 38 081990851+08 00 10 244 159 0 18 Jul 2023 10 28 38 +0800 PRI HTTP 2 0 400 150 0 0 001 65623457976afbcd3de55655954178b2 2023 07 18T10 28 38 498466789+08 00 10 244 159 0 18 Jul 2023 10 28 38 +0800 PRI HTTP 2 0 400 150 0 0 001 fb418e33264c0f6935a316b22457b1f6```

盛毅 · Answer

< David Espejo he him > this is some info i got from i deployed flyte on my local cluster do i need to change a ingress controller

David Espejo (he/him) · Answer

Sorry for the delay That s 1 2 instances I ve seen users having to switch from an nginx implementation due to gRPC issues Some users running also on local K8s cluster use Traefik or even MetaLB Besides the ingress portion for the PKCE flow to work properly with the `flyte core` chart implementation there are three native clients that have to be configured This is documented <https flyte 3837 org readthedocs build en 3837 deployment configuration auth setup html apply external auth server configuration|here>