https://flyte.org logo
Join Slack
Powered by
Hi Community, is there any simple approach to veri...
# flyte-deployment
x
Hi Community, is there any simple approach to verify the GRPC service of flyte admin works as expected? I tried to deploy 
flyte-core
helm chart on self-hosted kubernetes cluster but encounter certificate problem when trying to register a workflow remotely. The service is deployed with “Kubernetes Ingress Controller Fake Certificate” and all the ssl/tls related settings should be configured with default value of the template. I roughly looked through them, but did not find any obvious problem. BTW, the flyte console seems to work fine. When I try to 
flytectl register
with client config 
admin.insecure: false
(the default value by 
flytectl config init
), it complains about
Copy code
$ flytectl register files --project flytesnacks --domain development --archive flyte-package.tgz --version latest
 ------------------------------------------------------------------ -------- ----------------------------------------------------
| NAME                                                             | STATUS | ADDITIONAL INFO                                    |
 ------------------------------------------------------------------ -------- ----------------------------------------------------
| /tmp/register2617257857/0_flyte.workflows.example.say_hello_1.pb | Failed | Error registering file due to rpc error: code =    |
|                                                                  |        | Unavailable desc = connection error: desc =        |
|                                                                  |        | "transport: authentication handshake failed: x509: |
|                                                                  |        | "Kubernetes Ingress Controller Fake Certificate"   |
|                                                                  |        | certificate is not trusted"                        |
 ------------------------------------------------------------------ -------- ----------------------------------------------------
1 rows
Error: Connection Info: [Endpoint: dns:///flyte.XXX.com, InsecureConnection?: false, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: "Kubernetes Ingress Controller Fake Certificate" certificate is not trusted"
After changing the 
insecure
config to 
true
, the error message becomes
Copy code
$ flytectl register files --project flytesnacks --domain development --archive flyte-package.tgz --version latest
 ------------------------------------------------------------------ -------- ----------------------------------------------------
| NAME                                                             | STATUS | ADDITIONAL INFO                                    |
 ------------------------------------------------------------------ -------- ----------------------------------------------------
| /tmp/register3222452968/0_flyte.workflows.example.say_hello_1.pb | Failed | Error registering file due to rpc error: code =    |
|                                                                  |        | Unavailable desc = connection closed before server |
|                                                                  |        | preface received                                   |
 ------------------------------------------------------------------ -------- ----------------------------------------------------
1 rows
Error: Connection Info: [Endpoint: dns:///flyte.XXX.com, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection closed before server preface received
Actually, I am not sure the problem is caused by inappropriate client config or server settings. So I suppose the first step is to check the GRPC service of flyte admin. Just let me know if you have any comments. Thanks in advance.
k
How did you configure your Ingress and rules?
x
Hi @Katrina P, thanks for the prompt reply. the 
values.yaml
I use for ingress is listed as following:
Copy code
ingress:
    annotations:
      <http://haproxy-ingress.github.io/app-root|haproxy-ingress.github.io/app-root>: "/console"
      <http://haproxy-ingress.github.io/default-backend-redirect|haproxy-ingress.github.io/default-backend-redirect>: "/console"
      <http://ingress.kubernetes.io/cors-enable|ingress.kubernetes.io/cors-enable>: "true"
      <http://ingress.kubernetes.io/ssl-redirect|ingress.kubernetes.io/ssl-redirect>: "false"
      <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: haproxy
The only difference is that our kubernetes use haproxy ingress controller rather than nginx.
k
We use a different ingress controller, but just make sure HPAProxy is configured to proxy h2c requests -- haven't worked with HPAproxy but that's the first thing I'd verify. Second, I'd make sure you have the right ingress rules set up
kubectl get ingress -n namespaceName -o json
<-- something like this
x
Hi @Katrina P I paste the result to github gist temporarily: https://gist.github.com/huxuan/752252a10ee26addcf2fc3141977553b
BTW, I find a config 
insecureSkipVerify
so there is one more situation with 
insecure: false
and 
insecureSkipVerify: true
, and the error message becomes
Copy code
Error: rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type "text/plain; charset=utf-8"
k
Just took a look, I'm not 100% what the configuration is for HPAproxy but for traefik we have to change these rules for GRPC end points:
Copy code
- backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.AuthMetadataService/*
          pathType: ImplementationSpecific
basically just verify these are the right notations for your ingress
You can use something like https://github.com/fullstorydev/grpcurl to help test RPC methods/ endpoints
k
❤️ for @Katrina P - thank you for always helping the community
x
Hi @Katrina P, May I ask the specific change you made for traefik? I tried some configs for 
haproxy
[1], e.g., 
backend-protocal
, 
path-type
, but all failed. And everything is done by changing the annotations (I enabled the separate grpc ingress), I did not find any potential changes for the 
backend
part. [1] https://haproxy-ingress.github.io/v0.11/docs/configuration/keys
Finally, the problem is resolved! After looking through the log of haproxy (with one of my colleagues), it turns out 
flytectl
will request the port 443 even when 
insecure
is configured as 
true
. So I made a key mistake that disabled the 
tls
of the ingress which make the grpc service exposed at 80 port. And since we need to set a specific 
backend-protocol
, the separation of grpc ingress and normal http ingress is needed. In conclusion, for 
haproxy
ingress controller with self-signed certificate, the helm chart 
values.yaml
for ingress should be
Copy code
ingress:
    host: <http://example.com|example.com>
    separateGrpcIngress: true
    separateGrpcIngressAnnotations:
      <http://ingress.kubernetes.io/backend-protocol|ingress.kubernetes.io/backend-protocol>: "grpc"
    annotations:
      <http://ingress.kubernetes.io/app-root|ingress.kubernetes.io/app-root>: "/console"
      <http://ingress.kubernetes.io/default-backend-redirect|ingress.kubernetes.io/default-backend-redirect>: "/console"
      <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: haproxy
    tls:
      enabled: true
And for 
flytectl
client, the config should be
Copy code
admin:
  endpoint: dns:///example.com
  authType: Pkce
  insecure: false
  insecureSkipVerify: true
k
Cc @katrina / @Xuan Hu / @Samhita Alla we should just capture this somewhere for Others
k
@Xuan Hu Niice! glad you got it worked out.
250 Views
Open in Slack
PreviousNext
Powered by