盛毅
07/07/2023, 1:37 AMKevin Su
07/07/2023, 1:42 AM盛毅
07/07/2023, 1:46 AMKevin Su
07/07/2023, 1:48 AMinsecure_skip_verify: true
to admin config盛毅
07/07/2023, 1:54 AMKevin Su
07/11/2023, 6:28 AM盛毅
07/11/2023, 6:49 AMKevin Su
07/11/2023, 7:51 AM盛毅
07/11/2023, 7:53 AMDavid Espejo (he/him)
07/11/2023, 12:50 PMendpoint
?盛毅
07/12/2023, 1:29 AMadmin:
# For GRPC endpoints you might want to use dns:///flyte.myexample.com
endpoint: dns:///flyte.nginxbeta.k8s
#endpoint: dns:///10.24.73.16:32332
authType: Pkce
insecure: true
#insecure_skip_verify: true
logger:
show-source: true
level: 0
David Espejo (he/him)
07/12/2023, 11:08 AMadmin:
# For GRPC endpoints you might want to use dns:///flyte.myexample.com
endpoint: dns:///flyte-the-hard-way.uniondemo.run
authType: Pkce
insecure: false
insecureSkipVerify: true
logger:
show-source: true
level: 6
http1.x
error message typically refers to a gRPC connection using the wrong port. What are you using for Ingress? can you share a describe
of that resource?盛毅
07/13/2023, 2:41 AM[root@betam1 ~]# kubectl -n ingress-nginx get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.96.0.160 10.24.73.232 80:30973/TCP,443:30953/TCP 28m
ingress-nginx-controller-admission ClusterIP 10.96.3.112 <none> 443/TCP 35m
the ingress in flyte namespace
[root@betam1 ~]# kubectl -n ingress-nginx get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.96.0.160 10.24.73.232 80:30973/TCP,443:30953/TCP 28m
ingress-nginx-controller-admission ClusterIP 10.96.3.112 <none> 443/TCP 35m
the flyte-core ingress
[root@betam1 ~]# kubectl -n flyte describe ing flyte-core
Name: flyte-core
Labels: <none>
Namespace: flyte
Address: 10.24.73.21
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
flyte.nginxbeta.k8s
/console flyteconsole:80 (10.244.203.248:8080)
/console/* flyteconsole:80 (10.244.203.248:8080)
/api flyteadmin:80 (10.244.9.75:8088)
/api/* flyteadmin:80 (10.244.9.75:8088)
/healthcheck flyteadmin:80 (10.244.9.75:8088)
/v1/* flyteadmin:80 (10.244.9.75:8088)
/.well-known flyteadmin:80 (10.244.9.75:8088)
/.well-known/* flyteadmin:80 (10.244.9.75:8088)
/login flyteadmin:80 (10.244.9.75:8088)
/login/* flyteadmin:80 (10.244.9.75:8088)
/logout flyteadmin:80 (10.244.9.75:8088)
/logout/* flyteadmin:80 (10.244.9.75:8088)
/callback flyteadmin:80 (10.244.9.75:8088)
/callback/* flyteadmin:80 (10.244.9.75:8088)
/me flyteadmin:80 (10.244.9.75:8088)
/config flyteadmin:80 (10.244.9.75:8088)
/config/* flyteadmin:80 (10.244.9.75:8088)
/oauth2 flyteadmin:80 (10.244.9.75:8088)
/oauth2/* flyteadmin:80 (10.244.9.75:8088)
Annotations: <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 15m (x2 over 16m) nginx-ingress-controller Scheduled for sync
the ingress flyte-core-grpc
[root@betam1 ~]# kubectl -n flyte describe ing flyte-core-grpc
Name: flyte-core-grpc
Labels: <none>
Namespace: flyte
Address: 10.24.73.21
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
flyte.nginxbeta.k8s
/flyteidl.service.AdminService flyteadmin:81 (10.244.9.75:8089)
/flyteidl.service.AdminService/* flyteadmin:81 (10.244.9.75:8089)
/flyteidl.service.DataProxyService flyteadmin:81 (10.244.9.75:8089)
/flyteidl.service.DataProxyService/* flyteadmin:81 (10.244.9.75:8089)
/flyteidl.service.AuthMetadataService flyteadmin:81 (10.244.9.75:8089)
/flyteidl.service.AuthMetadataService/* flyteadmin:81 (10.244.9.75:8089)
/flyteidl.service.IdentityService flyteadmin:81 (10.244.9.75:8089)
/flyteidl.service.IdentityService/* flyteadmin:81 (10.244.9.75:8089)
/grpc.health.v1.Health flyteadmin:81 (10.244.9.75:8089)
/grpc.health.v1.Health/* flyteadmin:81 (10.244.9.75:8089)
/flyteidl.service.SignalService flyteadmin:81 (10.244.9.75:8089)
/flyteidl.service.SignalService/* flyteadmin:81 (10.244.9.75:8089)
Annotations: <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
<http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 15m (x2 over 16m) nginx-ingress-controller Scheduled for sync
the nginx-ingress has a loadbalance IP 10.24.73.232 , but in the ing describe above it got the address Address: 10.24.73.21
i have no idea what happended . but i can successfully visit the flyte console web ui using the ip 10.24.73.232David Espejo (he/him)
07/13/2023, 11:15 AM<http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
Aslo for auth, can you share your Helm values file?盛毅
07/14/2023, 1:43 AMauth:
grpcAuthorizationHeader: "flyte-authorization"
authorizedUris:
# This should point at your public http Uri.
- <http://flyte.nginxbeta.k8s>
# This will be used by internal services in the same namespace as flyteadmin
- <http://flyteadmin:80>
# This will be used by internal services in the same cluster but different namespaces
- <http://flyteadmin.flyte.svc.cluster.local:80> #assuming that flyte is installed in flyte namespace
# Controls app authentication config
appAuth:
authServerType: External
externalAuthServer:
allowedAudience: flytepropeller
baseUrl: "" # populate this value only if different from userauth baseUrl
metadataUrl: ".well-known/openid-configuration"
thirdPartyConfig:
flyteClient:
audience: flytepropeller
clientId: flytepropeller
redirectUri: <http://localhost:53593/callback>
scopes:
- all
- offline
- access_token
# Controls user authentication
userAuth:
redirectUrl: "<http://flyte.nginxbeta.k8s/console>" #add the flyte url
cookieSetting:
domain: "flyte.nginxbeta.k8s" # add here the sitename for flyte
sameSitePolicy: DefaultMode
openId:
baseUrl: "<http://10.24.50.130:31157/auth/realms/flyte>"
scopes:
- profile
- openid
- offline_access
clientId: flytecore
the following is part of the configmap flyte-admin-base-config
auth:
appAuth:
authServerType: External
externalAuthServer:
allowedAudience: flytepropeller
baseUrl: ""
metadataUrl: .well-known/openid-configuration
thirdPartyConfig:
flyteClient:
audience: flytepropeller
clientId: flytepropeller
redirectUri: <http://localhost:53593/callback>
scopes:
- all
- offline
- access_token
authorizedUris:
- <https://flyte.nginxbeta.k8s>
- <http://flyteadmin:80>
- <http://flyteadmin.flyte.svc.cluster.local:80>
grpcAuthorizationHeader: flyte-authorization
userAuth:
cookieSetting:
domain: flyte.nginxbeta.k8s
sameSitePolicy: DefaultMode
openId:
baseUrl: <http://10.24.50.130:31157/realms/flyte>
clientId: flytecore
scopes:
- profile
- openid
- offline_access
redirectUrl: <https://flyte.nginxbeta.k8s/console>
flyteadmin:
eventVersion: 2
metadataStoragePrefix:
- metadata
- admin
metricsScope: 'flyte:'
profilerPort: 10254
roleNameKey: <http://iam.amazonaws.com/role|iam.amazonaws.com/role>
testing:
host: <http://flyteadmin>
server:
grpcPort: 8089
httpPort: 8088
security:
allowCors: true
allowedHeaders:
- Content-Type
- flyte-authorization
allowedOrigins:
- '*'
secure: false
useAuth: true
David Espejo (he/him)
07/14/2023, 11:49 AMendpoint: dns:///flyte.nginxbeta.k8s:443
Also, for auth, please check out this guide (in review) to make sure it matches your deployment:
https://flyte--3837.org.readthedocs.build/en/3837/deployment/configuration/auth_setup.html盛毅
07/18/2023, 2:29 AM[root@gpu2 .flyte]# flytectl get projects
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2023-07-17T22:22:55-04:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2023-07-17T22:22:55-04:00"}
{"json":{"src":"viper.go:400"},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2023-07-17T22:22:55-04:00"}
{"json":{"src":"client.go:63"},"level":"info","msg":"Initialized Admin client","ts":"2023-07-17T22:22:55-04:00"}
{"json":{"src":"auth_interceptor.go:86"},"level":"debug","msg":"Request failed due to [rpc error: code = Unavailable desc = connection closed before server preface received]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2023-07-17T22:22:55-04:00"}
Error: Connection Info: [Endpoint: dns:///flyte.nginx.k8s:31821, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection closed before server preface received
{"json":{"src":"main.go:13"},"level":"error","msg":"Connection Info: [Endpoint: dns:///flyte.nginx.k8s:31821, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection closed before server preface received","ts":"2023-07-17T22:22:55-04:00"}
this is the log from nginx ingress controller,
2023-07-18T10:28:37.619817703+08:00 10.244.159.0 - - [18/Jul/2023:10:28:37 +0800] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.001 [] [] - - - - 9565a5c1e0480bab6e57defcec893209
2023-07-18T10:28:38.081990851+08:00 10.244.159.0 - - [18/Jul/2023:10:28:38 +0800] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.001 [] [] - - - - 65623457976afbcd3de55655954178b2
2023-07-18T10:28:38.498466789+08:00 10.244.159.0 - - [18/Jul/2023:10:28:38 +0800] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.001 [] [] - - - - fb418e33264c0f6935a316b22457b1f6
David Espejo (he/him)
07/19/2023, 12:05 PMflyte-core
chart implementation, there are three native clients that have to be configured. This is documented here