strong-barista-29632
12/14/2022, 1:12 PMflyte-core
helm chart on self-hosted kubernetes cluster but encounter certificate problem when trying to register a workflow remotely. The service is deployed with “Kubernetes Ingress Controller Fake Certificate” and all the ssl/tls related settings should be configured with default value of the template. I roughly looked through them, but did not find any obvious problem. BTW, the flyte console seems to work fine.
When I try to flytectl register
with client config admin.insecure: false
(the default value by flytectl config init
), it complains about
$ flytectl register files --project flytesnacks --domain development --archive flyte-package.tgz --version latest
------------------------------------------------------------------ -------- ----------------------------------------------------
| NAME | STATUS | ADDITIONAL INFO |
------------------------------------------------------------------ -------- ----------------------------------------------------
| /tmp/register2617257857/0_flyte.workflows.example.say_hello_1.pb | Failed | Error registering file due to rpc error: code = |
| | | Unavailable desc = connection error: desc = |
| | | "transport: authentication handshake failed: x509: |
| | | "Kubernetes Ingress Controller Fake Certificate" |
| | | certificate is not trusted" |
------------------------------------------------------------------ -------- ----------------------------------------------------
1 rows
Error: Connection Info: [Endpoint: dns:///flyte.XXX.com, InsecureConnection?: false, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: "Kubernetes Ingress Controller Fake Certificate" certificate is not trusted"
After changing the insecure
config to true
, the error message becomes
$ flytectl register files --project flytesnacks --domain development --archive flyte-package.tgz --version latest
------------------------------------------------------------------ -------- ----------------------------------------------------
| NAME | STATUS | ADDITIONAL INFO |
------------------------------------------------------------------ -------- ----------------------------------------------------
| /tmp/register3222452968/0_flyte.workflows.example.say_hello_1.pb | Failed | Error registering file due to rpc error: code = |
| | | Unavailable desc = connection closed before server |
| | | preface received |
------------------------------------------------------------------ -------- ----------------------------------------------------
1 rows
Error: Connection Info: [Endpoint: dns:///flyte.XXX.com, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection closed before server preface received
Actually, I am not sure the problem is caused by inappropriate client config or server settings. So I suppose the first step is to check the GRPC service of flyte admin.
Just let me know if you have any comments. Thanks in advance.limited-dog-47035
12/14/2022, 2:39 PMstrong-barista-29632
12/14/2022, 2:49 PMvalues.yaml
I use for ingress is listed as following:
ingress:
annotations:
<http://haproxy-ingress.github.io/app-root|haproxy-ingress.github.io/app-root>: "/console"
<http://haproxy-ingress.github.io/default-backend-redirect|haproxy-ingress.github.io/default-backend-redirect>: "/console"
<http://ingress.kubernetes.io/cors-enable|ingress.kubernetes.io/cors-enable>: "true"
<http://ingress.kubernetes.io/ssl-redirect|ingress.kubernetes.io/ssl-redirect>: "false"
<http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: haproxy
The only difference is that our kubernetes use haproxy ingress controller rather than nginx.limited-dog-47035
12/14/2022, 3:21 PMlimited-dog-47035
12/14/2022, 3:23 PMkubectl get ingress -n namespaceName -o json
<-- something like thisstrong-barista-29632
12/14/2022, 3:28 PMstrong-barista-29632
12/14/2022, 3:30 PMinsecureSkipVerify
so there is one more situation with insecure: false
and insecureSkipVerify: true
, and the error message becomes
Error: rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type "text/plain; charset=utf-8"
limited-dog-47035
12/14/2022, 3:31 PM- backend:
service:
name: flyteadmin
port:
number: 81
path: /flyteidl.service.AuthMetadataService/*
pathType: ImplementationSpecific
limited-dog-47035
12/14/2022, 3:32 PMlimited-dog-47035
12/14/2022, 3:36 PMfreezing-airport-6809
strong-barista-29632
12/15/2022, 7:45 AMhaproxy
[1], e.g., backend-protocal
, path-type
, but all failed. And everything is done by changing the annotations (I enabled the separate grpc ingress), I did not find any potential changes for the backend
part.
[1] https://haproxy-ingress.github.io/v0.11/docs/configuration/keysstrong-barista-29632
12/15/2022, 1:32 PMflytectl
will request the port 443 even when insecure
is configured as true
. So I made a key mistake that disabled the tls
of the ingress which make the grpc service exposed at 80 port. And since we need to set a specific backend-protocol
, the separation of grpc ingress and normal http ingress is needed.
In conclusion, for haproxy
ingress controller with self-signed certificate, the helm chart values.yaml
for ingress should be
ingress:
host: <http://example.com|example.com>
separateGrpcIngress: true
separateGrpcIngressAnnotations:
<http://ingress.kubernetes.io/backend-protocol|ingress.kubernetes.io/backend-protocol>: "grpc"
annotations:
<http://ingress.kubernetes.io/app-root|ingress.kubernetes.io/app-root>: "/console"
<http://ingress.kubernetes.io/default-backend-redirect|ingress.kubernetes.io/default-backend-redirect>: "/console"
<http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: haproxy
tls:
enabled: true
And for flytectl
client, the config should be
admin:
endpoint: dns:///example.com
authType: Pkce
insecure: false
insecureSkipVerify: true
freezing-airport-6809
limited-dog-47035
12/15/2022, 9:12 PM