https://flyte.org logo
Join Slack
Powered by
Hello, we are running an on-premise setup with th...
# ask-the-community
i
Hello, we are running an on-premise setup with the latest flyte-binary version based on the instructions in this document. Port-forwards to http and grpc work successfully, but I have problems with the setup of nginx-ingress. This is the ingress section of the values file:
Copy code
ingress:
  create: true
  ingressClassName: nginx
  # --- Ingress annotations applied to both HTTP and GRPC ingresses.
  commonAnnotations:
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
    <http://ingress.kubernetes.io/rewrite-target|ingress.kubernetes.io/rewrite-target>: /
    <http://nginx.ingress.kubernetes.io/proxy-body-size|nginx.ingress.kubernetes.io/proxy-body-size>: "0"
    <http://nginx.ingress.kubernetes.io/proxy-read-timeout|nginx.ingress.kubernetes.io/proxy-read-timeout>: "600"
    <http://nginx.ingress.kubernetes.io/proxy-send-timeout|nginx.ingress.kubernetes.io/proxy-send-timeout>: "600"
    <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
  httpAnnotations:
    <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
  grpcAnnotations:
    <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
  host: <internal hostname>
  #  --- TLS Settings
  tls:
    - hosts:
      - <internal hostname>
      secretName: mlflyte
GRPC port-forwarding works successfully:
Copy code
grpcurl --plaintext -v localhost:8089 list
flyteidl.service.AdminService
flyteidl.service.DataProxyService
flyteidl.service.SignalService
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection
However, grpcurl doesn't return when connecting to the ingress controller. 
grpcurl -vv <internal hostname>:443 list
The reflection request returns status 499 in the log records of the ingress controller pod.
Copy code
kubectl logs -n ingress-nginx  ingress-nginx-controller-76df688779-cbjvk
10.42.0.1 - - [04/Jan/2024:15:10:22 +0000] "POST /grpc.reflection.v1.ServerReflection/ServerReflectionInfo HTTP/2.0" 499 0 "-" "grpcurl/v1.8.9 grpc-go/1.57.0" 16 1.928 [flyte-minio-9200] [] - - - - 86be7a20ac12cee2b6599b8fc00fe94a
Any ideas are kindly appreciated.
d
Hi Ingo and welcome to the Flyte community What's the status of the SSL certificate you're using? Is it CA signed?
y
could you also paste here the 
kubectl get -o yaml
dump (redacted if you need) of the relevant ingress objects and services?
i
Yes, it's signed by our internal CA and the chain (root/intermediate/server) is included in the tls secret (secretName: mlflyte)
Copy code
kubectl get ingress flyte-binary-grpc -n flyte -o yaml
apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: Ingress
metadata:
  annotations:
    <http://ingress.kubernetes.io/rewrite-target|ingress.kubernetes.io/rewrite-target>: /
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
    <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte-binary
    <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
    <http://nginx.ingress.kubernetes.io/affinity|nginx.ingress.kubernetes.io/affinity>: cookie
    <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
    <http://nginx.ingress.kubernetes.io/proxy-body-size|nginx.ingress.kubernetes.io/proxy-body-size>: "0"
    <http://nginx.ingress.kubernetes.io/proxy-buffer-size|nginx.ingress.kubernetes.io/proxy-buffer-size>: 32k
    <http://nginx.ingress.kubernetes.io/proxy-read-timeout|nginx.ingress.kubernetes.io/proxy-read-timeout>: "600"
    <http://nginx.ingress.kubernetes.io/proxy-send-timeout|nginx.ingress.kubernetes.io/proxy-send-timeout>: "600"
    <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
  creationTimestamp: "2024-01-03T08:19:41Z"
  generation: 1
  labels:
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: flyte-binary
    <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: flyte-binary
    <http://app.kubernetes.io/version|app.kubernetes.io/version>: 1.16.0
    <http://helm.sh/chart|helm.sh/chart>: flyte-binary-v1.10.6
  name: flyte-binary-grpc
  namespace: flyte
  resourceVersion: "24598140"
  uid: 250a6f4c-e343-4263-8158-792bdb28d642
spec:
  ingressClassName: nginx
  rules:
  - host: mlflyte.accounts.intern
    http:
      paths:
      - backend:
          service:
            name: flyte-binary-grpc
            port:
              number: 8089
        path: /flyteidl.service.AdminService
        pathType: ImplementationSpecific
      - backend:
          service:
            name: flyte-binary-grpc
            port:
              number: 8089
        path: /flyteidl.service.AdminService/*
        pathType: ImplementationSpecific
      - backend:
          service:
            name: flyte-binary-grpc
            port:
              number: 8089
        path: /flyteidl.service.DataProxyService
        pathType: ImplementationSpecific
      - backend:
          service:
            name: flyte-binary-grpc
            port:
              number: 8089
        path: /flyteidl.service.DataProxyService/*
        pathType: ImplementationSpecific
      - backend:
          service:
            name: flyte-binary-grpc
            port:
              number: 8089
        path: /flyteidl.service.AuthMetadataService
        pathType: ImplementationSpecific
      - backend:
          service:
            name: flyte-binary-grpc
            port:
              number: 8089
        path: /flyteidl.service.AuthMetadataService/*
        pathType: ImplementationSpecific
      - backend:
          service:
            name: flyte-binary-grpc
            port:
              number: 8089
        path: /flyteidl.service.IdentityService
        pathType: ImplementationSpecific
      - backend:
          service:
            name: flyte-binary-grpc
            port:
              number: 8089
        path: /flyteidl.service.IdentityService/*
        pathType: ImplementationSpecific
      - backend:
          service:
            name: flyte-binary-grpc
            port:
              number: 8089
        path: /grpc.health.v1.Health
        pathType: ImplementationSpecific
      - backend:
          service:
            name: flyte-binary-grpc
            port:
              number: 8089
        path: /grpc.health.v1.Health/*
        pathType: ImplementationSpecific
      - backend:
          service:
            name: flyte-binary-grpc
            port:
              number: 8089
        path: /flyteidl.service.SignalService
        pathType: ImplementationSpecific
      - backend:
          service:
            name: flyte-binary-grpc
            port:
              number: 8089
        path: /flyteidl.service.SignalService/*
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - <internal hostname>
    secretName: mlflyte
status:
  loadBalancer:
    ingress:
    - ip: 10....
y
I assume that of the various server reflection settings in flyte you have them all on? (which is how it’s working with port-forward I assume). i assume then this is something with the nginx ingress layer. can you try hitting the ingress with something like flytectl or pyflyte or (flyte-cli, which is deprecated but still correctly invokes the grpc client)?
i don’t see anything that stands out in the ingress
if you run into issues calling it with flytectl, can also bump up logging level in its config
Copy code
logger:
  show-source: true
  level: 5
i
If I use the port-forward without TLS everything seems to work (at least in IPv4)
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: localhost:8089
  authType: Pkce
  insecure: true
logger:
  show-source: true
  level: 6
I can use the Web UI and see the results. There's an error message related to IPv6, but I don't know if this matters.
Copy code
Running Execution on Remote.
Handling connection for 8089

[✔] Go to <http://localhost:8089/console/projects/flytesnacks/domains/development/executions/f3b95738f187d45afa16> to see execution in the console.
E0105 12:48:53.227532 3205487 portforward.go:394] error copying from local connection to remote stream: read tcp6 [::1]:8089->[::1]:52402: read: connection reset by peer
I use the following config for grpc ingress
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: internalhost:443
  authType: Pkce
  insecure: false
  insecureSkipVerify: true
logger:
  show-source: true
  level: 6
Getting this result:
Copy code
Running Execution on Remote.
E0105 12:55:33.197051907 3211812 ssl_transport_security.cc:1495] Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.
Failed with Exception Code: SYSTEM:Unknown
RPC Failed, with Status: StatusCode.UNAVAILABLE
        details: failed to connect to all addresses; last error: UNKNOWN: Ssl handshake failed
        Debug string UNKNOWN:Failed to pick subchannel {created_time:"2024-01-05T12:55:33.198452094+01:00", children:[UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: Ssl handshake failed {grpc_status:14, created_time:"2024-01-05T12:55:33.198449379+01:00"}]}
Interesting. flytectl shows more detailed errors:
Copy code
flytectl create project --name ikproject01 --id ikproj01 --description "ik 1st test project" --labels app=ikproj01
{"json":{"src":"client.go:63"},"level":"info","msg":"Initialized Admin client","ts":"2024-01-05T13:15:42+01:00"}
{"json":{"src":"auth_interceptor.go:86"},"level":"debug","msg":"Request failed due to [rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: http2: frame too large\"]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2024-01-05T13:15:42+01:00"}
Error: Connection Info: [Endpoint: mlflyte.accounts.intern:80, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: http2: frame too large"
{"json":{"src":"main.go:13"},"level":"error","msg":"Connection Info: [Endpoint: mlflyte.accounts.intern:80, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: http2: frame too large\"","ts":"2024-01-05T13:15:42+01:00"}
There's a nice posting and explanation of the "http2: frame too large" message. Kenneth Jenkins explains why this happens and how it can be fixed (by adding plaintext HTTP/2 support using the h2c package). Is this an option for the Flyte binary package?
d
have you tried using 
insecure: true
?
i
If I use 
insecure: true
with port 80 I will get the following error message. It's the same message regardless if I set insecureSkipVerify true or false.
Copy code
Running Execution on Remote.
Failed with Exception Code: SYSTEM:Unknown
RPC Failed, with Status: StatusCode.UNAVAILABLE
        details: failed to connect to all addresses; last error: INTERNAL: Trying to connect an http1.x server
        Debug string UNKNOWN:Failed to pick subchannel {created_time:"2024-01-08T08:59:05.109662197+01:00", children:[UNKNOWN:failed to connect to all addresses; last error: INTERNAL: Trying to connect an http1.x server {grpc_status:14, created_time:"2024-01-08T08:59:05.109658711+01:00"}]}
Garret's hint helped me to fix the issue.
15 Views
Open in Slack
PreviousNext
Powered by