https://flyte.org logo
Join Slack
Powered by
hi, i deployed flyte to my local k8s cluster, i ca...
# ask-the-community
u
hi, i deployed flyte to my local k8s cluster, i can view the web ui, but can not use the command line to execute workflows, the following are is the error, any way to workaround,thank you!
k
did you set insecure in flytectl_config to false?
u
this is the config.yaml
k
try to add 
insecure_skip_verify: true
to admin config
u
@Kevin Su hi kevin, same error
@Kevin Su seems like the command line used https://flyte.nginxbeta.k8s, but actually it is http://flyte.nginxbeta.k8s
k
how did you install flyte? use helm install?
u
helm install
k
cc @David Espejo (he/him) did you run into this issue before?
wait, could you set insecure to true instead?
u
already set it true, still got the same error
d
hey @*盛毅* what chart did you install*?*
have you tried not specifying the port in 
endpoint
?
u
@David Espejo (he/him) hello David, i used the flyte-core helm chart and wanna test it in multi-cluster, the error message showed rpc request to a http server, how to config the endpoint to use http protocol,here are my config.yaml . Can you show me your config file???? thank you! 
admin:
# For GRPC endpoints you might want to use dns:///flyte.myexample.com
endpoint: dns:///flyte.nginxbeta.k8s
#endpoint: dns:///10.24.73.16:32332
authType: Pkce
insecure: true
#insecure_skip_verify: true
logger:
show-source: true
level: 0
d
sure, this is my config file:
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///flyte-the-hard-way.uniondemo.run
  authType: Pkce
  insecure: false  
  insecureSkipVerify: true

logger:
  show-source: true
  level: 6
the 
http1.x
error message typically refers to a gRPC connection using the wrong port. What are you using for Ingress? can you share a 
describe
of that resource?
u
the ingress-nginx 
[root@betam1 ~]# kubectl -n ingress-nginx get svc
NAME                                 TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.96.0.160   10.24.73.232   80:30973/TCP,443:30953/TCP   28m
ingress-nginx-controller-admission   ClusterIP      10.96.3.112   <none>         443/TCP                      35m
the ingress in flyte namespace 
[root@betam1 ~]# kubectl -n ingress-nginx get svc
NAME                                 TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.96.0.160   10.24.73.232   80:30973/TCP,443:30953/TCP   28m
ingress-nginx-controller-admission   ClusterIP      10.96.3.112   <none>         443/TCP                      35m
the flyte-core ingress 
[root@betam1 ~]# kubectl -n flyte describe ing flyte-core
Name:             flyte-core
Labels:           <none>
Namespace:        flyte
Address:          10.24.73.21
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host                 Path  Backends
----                 ----  --------
flyte.nginxbeta.k8s
/console         flyteconsole:80 (10.244.203.248:8080)
/console/*       flyteconsole:80 (10.244.203.248:8080)
/api             flyteadmin:80 (10.244.9.75:8088)
/api/*           flyteadmin:80 (10.244.9.75:8088)
/healthcheck     flyteadmin:80 (10.244.9.75:8088)
/v1/*            flyteadmin:80 (10.244.9.75:8088)
/.well-known     flyteadmin:80 (10.244.9.75:8088)
/.well-known/*   flyteadmin:80 (10.244.9.75:8088)
/login           flyteadmin:80 (10.244.9.75:8088)
/login/*         flyteadmin:80 (10.244.9.75:8088)
/logout          flyteadmin:80 (10.244.9.75:8088)
/logout/*        flyteadmin:80 (10.244.9.75:8088)
/callback        flyteadmin:80 (10.244.9.75:8088)
/callback/*      flyteadmin:80 (10.244.9.75:8088)
/me              flyteadmin:80 (10.244.9.75:8088)
/config          flyteadmin:80 (10.244.9.75:8088)
/config/*        flyteadmin:80 (10.244.9.75:8088)
/oauth2          flyteadmin:80 (10.244.9.75:8088)
/oauth2/*        flyteadmin:80 (10.244.9.75:8088)
Annotations:           <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
Events:
Type    Reason  Age                From                      Message
----    ------  ----               ----                      -------
Normal  Sync    15m (x2 over 16m)  nginx-ingress-controller  Scheduled for sync
the ingress flyte-core-grpc 
[root@betam1 ~]# kubectl -n flyte describe ing flyte-core-grpc
Name:             flyte-core-grpc
Labels:           <none>
Namespace:        flyte
Address:          10.24.73.21
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host                 Path  Backends
----                 ----  --------
flyte.nginxbeta.k8s
/flyteidl.service.AdminService            flyteadmin:81 (10.244.9.75:8089)
/flyteidl.service.AdminService/*          flyteadmin:81 (10.244.9.75:8089)
/flyteidl.service.DataProxyService        flyteadmin:81 (10.244.9.75:8089)
/flyteidl.service.DataProxyService/*      flyteadmin:81 (10.244.9.75:8089)
/flyteidl.service.AuthMetadataService     flyteadmin:81 (10.244.9.75:8089)
/flyteidl.service.AuthMetadataService/*   flyteadmin:81 (10.244.9.75:8089)
/flyteidl.service.IdentityService         flyteadmin:81 (10.244.9.75:8089)
/flyteidl.service.IdentityService/*       flyteadmin:81 (10.244.9.75:8089)
/grpc.health.v1.Health                    flyteadmin:81 (10.244.9.75:8089)
/grpc.health.v1.Health/*                  flyteadmin:81 (10.244.9.75:8089)
/flyteidl.service.SignalService           flyteadmin:81 (10.244.9.75:8089)
/flyteidl.service.SignalService/*         flyteadmin:81 (10.244.9.75:8089)
Annotations:           <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
<http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
Events:
Type    Reason  Age                From                      Message
----    ------  ----               ----                      -------
Normal  Sync    15m (x2 over 16m)  nginx-ingress-controller  Scheduled for sync
the nginx-ingress has a loadbalance IP 10.24.73.232 , but in the ing describe above it got the address 
Address:          10.24.73.21
i have no idea what happended . but i can successfully visit the flyte console web ui using the ip 10.24.73.232
@Kevin Su @David Espejo (he/him) got those errors after i enabled auth section in the helm chart, before that it is ok!
d
I don't see this annotation in your ingress resource 
<http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
Aslo for auth, can you share your Helm values file?
u
the auth part from helm values file , and i deployed on my local k8s cluster! 
auth:
grpcAuthorizationHeader: "flyte-authorization"
authorizedUris:
# This should point at your public http Uri.
- <http://flyte.nginxbeta.k8s>
# This will be used by internal services in the same namespace as flyteadmin
- <http://flyteadmin:80>
# This will be used by internal services in the same cluster but different namespaces
- <http://flyteadmin.flyte.svc.cluster.local:80> #assuming that flyte is installed in flyte namespace
# Controls app authentication config
appAuth:
authServerType: External
externalAuthServer:
allowedAudience: flytepropeller
baseUrl: "" # populate this value only if different from userauth baseUrl
metadataUrl: ".well-known/openid-configuration"
thirdPartyConfig:
flyteClient:
audience: flytepropeller
clientId: flytepropeller
redirectUri: <http://localhost:53593/callback>
scopes:
- all
- offline
- access_token
# Controls user authentication
userAuth:
redirectUrl: "<http://flyte.nginxbeta.k8s/console>" #add the flyte url
cookieSetting:
domain: "flyte.nginxbeta.k8s" # add here the sitename for flyte
sameSitePolicy: DefaultMode
openId:
baseUrl: "<http://10.24.50.130:31157/auth/realms/flyte>"
scopes:
- profile
- openid
- offline_access
clientId: flytecore
the following is part of the configmap flyte-admin-base-config
Copy code
auth:
  appAuth:
    authServerType: External
    externalAuthServer:
      allowedAudience: flytepropeller
      baseUrl: ""
      metadataUrl: .well-known/openid-configuration
    thirdPartyConfig:
      flyteClient:
        audience: flytepropeller
        clientId: flytepropeller
        redirectUri: <http://localhost:53593/callback>
        scopes:
        - all
        - offline
        - access_token
  authorizedUris:
  - <https://flyte.nginxbeta.k8s>
  - <http://flyteadmin:80>
  - <http://flyteadmin.flyte.svc.cluster.local:80>
  grpcAuthorizationHeader: flyte-authorization
  userAuth:
    cookieSetting:
      domain: flyte.nginxbeta.k8s
      sameSitePolicy: DefaultMode
    openId:
      baseUrl: <http://10.24.50.130:31157/realms/flyte>
      clientId: flytecore
      scopes:
      - profile
      - openid
      - offline_access
    redirectUrl: <https://flyte.nginxbeta.k8s/console>
flyteadmin:
  eventVersion: 2
  metadataStoragePrefix:
  - metadata
  - admin
  metricsScope: 'flyte:'
  profilerPort: 10254
  roleNameKey: <http://iam.amazonaws.com/role|iam.amazonaws.com/role>
  testing:
    host: <http://flyteadmin>
server:
  grpcPort: 8089
  httpPort: 8088
  security:
    allowCors: true
    allowedHeaders:
    - Content-Type
    - flyte-authorization
    allowedOrigins:
    - '*'
    secure: false
    useAuth: true
d
in some cases, if redirects are not working (because there's some networking component in the middle preventing it), specifying the port in config helps, like: 
endpoint: dns:///flyte.nginxbeta.k8s:443
Also, for auth, please check out this guide (in review) to make sure it matches your deployment: https://flyte--3837.org.readthedocs.build/en/3837/deployment/configuration/auth_setup.html
u
@David Espejo (he/him) hi David, sorry to interrupt you again, this is what i got when using flytectl , i can visit the web ui ,cannot connect using cli, is it something wrong the nginx ingess???
Copy code
[root@gpu2 .flyte]# flytectl get projects
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2023-07-17T22:22:55-04:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2023-07-17T22:22:55-04:00"}
{"json":{"src":"viper.go:400"},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2023-07-17T22:22:55-04:00"}
{"json":{"src":"client.go:63"},"level":"info","msg":"Initialized Admin client","ts":"2023-07-17T22:22:55-04:00"}
{"json":{"src":"auth_interceptor.go:86"},"level":"debug","msg":"Request failed due to [rpc error: code = Unavailable desc = connection closed before server preface received]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2023-07-17T22:22:55-04:00"}
Error: Connection Info: [Endpoint: dns:///flyte.nginx.k8s:31821, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection closed before server preface received
{"json":{"src":"main.go:13"},"level":"error","msg":"Connection Info: [Endpoint: dns:///flyte.nginx.k8s:31821, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection closed before server preface received","ts":"2023-07-17T22:22:55-04:00"}
this is the log from nginx ingress controller,
Copy code
2023-07-18T10:28:37.619817703+08:00 10.244.159.0 - - [18/Jul/2023:10:28:37 +0800] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.001 [] [] - - - - 9565a5c1e0480bab6e57defcec893209

2023-07-18T10:28:38.081990851+08:00 10.244.159.0 - - [18/Jul/2023:10:28:38 +0800] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.001 [] [] - - - - 65623457976afbcd3de55655954178b2

2023-07-18T10:28:38.498466789+08:00 10.244.159.0 - - [18/Jul/2023:10:28:38 +0800] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.001 [] [] - - - - fb418e33264c0f6935a316b22457b1f6
@David Espejo (he/him) this is some info i got from https://discuss.flyte.org/t/10376700/hello-could-somebody-help-me-debug-connection-to-the-flyte-c , i deployed flyte on my local cluster, do i need to change a ingress controller???
d
Sorry for the delay That's 1/2 instances I've seen users having to switch from an nginx implementation due to gRPC issues. Some users running also on local K8s cluster, use Traefik or even MetaLB. Besides the ingress portion, for the PKCE flow to work properly with the 
flyte-core
chart implementation, there are three native clients that have to be configured. This is documented here
22 Views
Open in Slack
PreviousNext
Powered by