https://flyte.org logo
#flyte-deployment
Title
# flyte-deployment
x

Xuan Hu

12/14/2022, 1:12 PM
Hi Community, is there any simple approach to verify the GRPC service of flyte admin works as expected? I tried to deploy
flyte-core
helm chart on self-hosted kubernetes cluster but encounter certificate problem when trying to register a workflow remotely. The service is deployed with “Kubernetes Ingress Controller Fake Certificate” and all the ssl/tls related settings should be configured with default value of the template. I roughly looked through them, but did not find any obvious problem. BTW, the flyte console seems to work fine. When I try to
flytectl register
with client config
admin.insecure: false
(the default value by
flytectl config init
), it complains about
Copy code
$ flytectl register files --project flytesnacks --domain development --archive flyte-package.tgz --version latest
 ------------------------------------------------------------------ -------- ----------------------------------------------------
| NAME                                                             | STATUS | ADDITIONAL INFO                                    |
 ------------------------------------------------------------------ -------- ----------------------------------------------------
| /tmp/register2617257857/0_flyte.workflows.example.say_hello_1.pb | Failed | Error registering file due to rpc error: code =    |
|                                                                  |        | Unavailable desc = connection error: desc =        |
|                                                                  |        | "transport: authentication handshake failed: x509: |
|                                                                  |        | "Kubernetes Ingress Controller Fake Certificate"   |
|                                                                  |        | certificate is not trusted"                        |
 ------------------------------------------------------------------ -------- ----------------------------------------------------
1 rows
Error: Connection Info: [Endpoint: dns:///flyte.XXX.com, InsecureConnection?: false, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: "Kubernetes Ingress Controller Fake Certificate" certificate is not trusted"
After changing the
insecure
config to
true
, the error message becomes
Copy code
$ flytectl register files --project flytesnacks --domain development --archive flyte-package.tgz --version latest
 ------------------------------------------------------------------ -------- ----------------------------------------------------
| NAME                                                             | STATUS | ADDITIONAL INFO                                    |
 ------------------------------------------------------------------ -------- ----------------------------------------------------
| /tmp/register3222452968/0_flyte.workflows.example.say_hello_1.pb | Failed | Error registering file due to rpc error: code =    |
|                                                                  |        | Unavailable desc = connection closed before server |
|                                                                  |        | preface received                                   |
 ------------------------------------------------------------------ -------- ----------------------------------------------------
1 rows
Error: Connection Info: [Endpoint: dns:///flyte.XXX.com, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection closed before server preface received
Actually, I am not sure the problem is caused by inappropriate client config or server settings. So I suppose the first step is to check the GRPC service of flyte admin. Just let me know if you have any comments. Thanks in advance.
k

Katrina P

12/14/2022, 2:39 PM
How did you configure your Ingress and rules?
x

Xuan Hu

12/14/2022, 2:49 PM
Hi @Katrina P, thanks for the prompt reply. the
values.yaml
I use for ingress is listed as following:
Copy code
ingress:
    annotations:
      <http://haproxy-ingress.github.io/app-root|haproxy-ingress.github.io/app-root>: "/console"
      <http://haproxy-ingress.github.io/default-backend-redirect|haproxy-ingress.github.io/default-backend-redirect>: "/console"
      <http://ingress.kubernetes.io/cors-enable|ingress.kubernetes.io/cors-enable>: "true"
      <http://ingress.kubernetes.io/ssl-redirect|ingress.kubernetes.io/ssl-redirect>: "false"
      <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: haproxy
The only difference is that our kubernetes use haproxy ingress controller rather than nginx.
k

Katrina P

12/14/2022, 3:21 PM
We use a different ingress controller, but just make sure HPAProxy is configured to proxy h2c requests -- haven't worked with HPAproxy but that's the first thing I'd verify. Second, I'd make sure you have the right ingress rules set up
kubectl get ingress -n namespaceName -o json
<-- something like this
x

Xuan Hu

12/14/2022, 3:28 PM
Hi @Katrina P I paste the result to github gist temporarily: https://gist.github.com/huxuan/752252a10ee26addcf2fc3141977553b
BTW, I find a config
insecureSkipVerify
so there is one more situation with
insecure: false
and
insecureSkipVerify: true
, and the error message becomes
Copy code
Error: rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type "text/plain; charset=utf-8"
k

Katrina P

12/14/2022, 3:31 PM
Just took a look, I'm not 100% what the configuration is for HPAproxy but for traefik we have to change these rules for GRPC end points:
Copy code
- backend:
            service:
              name: flyteadmin
              port:
                number: 81
          path: /flyteidl.service.AuthMetadataService/*
          pathType: ImplementationSpecific
basically just verify these are the right notations for your ingress
You can use something like https://github.com/fullstorydev/grpcurl to help test RPC methods/ endpoints
k

Ketan (kumare3)

12/14/2022, 3:55 PM
❤️ for @Katrina P - thank you for always helping the community
x

Xuan Hu

12/15/2022, 7:45 AM
Hi @Katrina P, May I ask the specific change you made for traefik? I tried some configs for
haproxy
[1], e.g.,
backend-protocal
,
path-type
, but all failed. And everything is done by changing the annotations (I enabled the separate grpc ingress), I did not find any potential changes for the
backend
part. [1] https://haproxy-ingress.github.io/v0.11/docs/configuration/keys
Finally, the problem is resolved! After looking through the log of haproxy (with one of my colleagues), it turns out
flytectl
will request the port 443 even when
insecure
is configured as
true
. So I made a key mistake that disabled the
tls
of the ingress which make the grpc service exposed at 80 port. And since we need to set a specific
backend-protocol
, the separation of grpc ingress and normal http ingress is needed. In conclusion, for
haproxy
ingress controller with self-signed certificate, the helm chart
values.yaml
for ingress should be
Copy code
ingress:
    host: <http://example.com|example.com>
    separateGrpcIngress: true
    separateGrpcIngressAnnotations:
      <http://ingress.kubernetes.io/backend-protocol|ingress.kubernetes.io/backend-protocol>: "grpc"
    annotations:
      <http://ingress.kubernetes.io/app-root|ingress.kubernetes.io/app-root>: "/console"
      <http://ingress.kubernetes.io/default-backend-redirect|ingress.kubernetes.io/default-backend-redirect>: "/console"
      <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: haproxy
    tls:
      enabled: true
And for
flytectl
client, the config should be
Copy code
admin:
  endpoint: dns:///example.com
  authType: Pkce
  insecure: false
  insecureSkipVerify: true
k

Ketan (kumare3)

12/15/2022, 4:09 PM
Cc @katrina / @Xuan Hu / @Samhita Alla we should just capture this somewhere for Others
k

Katrina P

12/15/2022, 9:12 PM
@Xuan Hu Niice! glad you got it worked out.
22 Views