https://flyte.org
Join Slack
[RESOLVED] Hey all, I’m having a bit of trouble with the guide for multi cluster deployment, in part...
# flyte-support
h
[RESOLVED] Hey all, I’m having a bit of trouble with the guide for multi cluster deployment, in particular when trying to deploy the control plane with the cluster config. I’m receiving errors in the init containers for the flyteadmin.
💯 3
✅ 1
Right now, the error in 
sync-cluster-resources
init container is
Copy code
panic: unable to load root certificates: unable to parse bytes as PEM block

goroutine 1 [running]:
main.main()
	/go/src/github.com/flyteorg/flyteadmin/cmd/main.go:13 +0x91
I set up the secret according to the guide. I did it both in plain text and encoded.
One error that I ran into with the chart is in the 
values-override.yaml
file, it uses this config:
Copy code
flyteadmin:
  additionalVolumes:
  - name: cluster-credentials
    secret:
      secretName: cluster-credentials
  additionalVolumeMounts:
  - name: cluster-credentials
    mountPath: /var/run/credentials
but 
additionalVolumeMounts
will only mount to the main container. so 
sync-cluster-resources
isn’t getting the cluster credentials file mounted in the current chart. That’s an easy fix and so now I’m stuck at the PEM error.
If anyone has any guidance on this, please let me know.
It appears to be something to do with how I’m storing the text block on AWS Secrets manager and not having a trailing newline. I switched the cluster config to just use the 
flyteadmin-token
(starting with data plane in the same cluster before adding a second cluster) and the PEM block error went away.
Plain text format of the secret on AWS SM:
Copy code
{
  "cacert": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----\n",
  "token": "..."
}
thx 3
f
hey Alex, great to meet you. interesting that you are deploying a multi-cluster flyte
would you be open to talking and sharing the usecase 😄
h
We run separate clusters for some environments and this looks like a nice way to control the isolation. Unfortunately, I can’t even connect with 
flytectl
right now due to some mysterious grpc errors:
Copy code
Error: Connection Info: [Endpoint: dns:///..., InsecureConnection?: false, AuthMode: Pkce]: rpc error: code = Unknown desc = unexpected HTTP status code received from server: 464 (); malformed header: missing HTTP content-type
Copy code
Error: Connection Info: [Endpoint: dns:///..., InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection closed before server preface received
Not sure what I’m going to do since the docs make it seem easy and it’s just not working. I might try the simple flyte setup and then abandon it if 
flytectl
still doesn’t work. It worked ~1 month ago but no longer.
lol, yea, 
flytectl
still not working with the simple binary deployment and port-forwarding just like in the deployment docs.
Copy code
╰─❯ flytectl get projects                                
Error: Connection Info: [Endpoint: dns:///localhost:8088, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection closed before server preface received
f
Cc @freezing-boots-56761 ?
Seems like a proxy error? Odd
@hallowed-doctor-67759 sorry for the trouble will take a look at this tomorrow morning
Should the port be 8089
What port are you forwarding
h
Copy code
─❯ kubectl -n flyte port-forward service/flyte-backend-flyte-binary 8088:8088 8089:8089
Forwarding from 127.0.0.1:8088 -> 8088
Forwarding from [::1]:8088 -> 8088
Forwarding from 127.0.0.1:8089 -> 8089
Forwarding from [::1]:8089 -> 8089
Handling connection for 8088
guide says 8088. I just tried 8089. huh, now it works. didn’t before.
hmmm, I wonder why it wasn’t working on the 
flyte-core
deployment
f
Ohh no - cc @thankful-minister-83577 @hallowed-doctor-67759 we changed to single binary deployment for Simplification very recently
Sorry for the trouble
h
f
Flyte-core is not single binary and port forwarding will jot work for Ui
This is why we moved to a monolith - but you can always split up later
h
so I needed to add 
--insecure
and hadn’t discovered that yet:
Copy code
╰─❯ flytectl config init --host localhost:8089
This action will overwrite an existing config file at [/Users/alexifm/.flyte/config.yaml]. Do you want to continue? [y/n]: y
Init flytectl config file at [/Users/alexifm/.flyte/config.yaml]% 
╰─❯ flytectl get projects                     
Error: Connection Info: [Endpoint: dns:///localhost:8089, InsecureConnection?: false, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"
Flyte-core is not single binary and port forwarding will jot work for Ui
right, I got the ingress working and can visit the console via our DNS name but the connection to GRPC via flytectl wasn’t working
Okay, got the multi cluster deploy working and using 
kubefwd
to get 
flytectl
working since DNS isn’t.
Copy code
╰─❯ sudo -E kubefwd svc -n flyte -f metadata.name=flyteadmin
Copy code
╰─❯ cat /etc/hosts
───────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: /etc/hosts
───────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  ...
  12   │ 127.1.27.1       flyteadmin flyteadmin.flyte flyteadmin-flyte flyteadmin.flyte.svc flyteadmin-flyte-svc flytead
       │ min.flyte.svc.cluster.local flyteadmin-flyte-svc-cluster-local flyteadmin.flyte.infima-staging-eks flyteadmin-f
       │ lyte-infima-staging-eks flyteadmin.flyte.svc.infima-staging-eks flyteadmin-flyte-svc-infima-staging-eks flytead
       │ min.flyte.svc.cluster.infima-staging-eks flyteadmin-flyte-svc-cluster-infima-staging-eks
───────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Copy code
╰─❯ flytectl config init --host flyteadmin:81 --insecure
@freezing-airport-6809 our issue with 
flytectl
is due to the ingress controller not supporting GRPC. 🤷 We’re going to switch back to the original setup we used from the old version of the docs.
f
interesting, please leave a comment, we will keep the original setup always as well
cc @thankful-minister-83577 / @freezing-boots-56761 just a note
t
sorry haven’t caught up with the thread yet… what’s not working with grpc?
the ingress in the flyte-binary helm chart should work with grpc
h
I didn’t set up ingress with the flyte-binary. I was setting up ingress with flyte-core and the controller (skipper) we use elsewhere couldn’t be used with flyte due to lack of grpc support.
164 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by