Alex Papanicolaou
02/14/2023, 4:58 PMsync-cluster-resources
init container is
panic: unable to load root certificates: unable to parse bytes as PEM block
goroutine 1 [running]:
main.main()
/go/src/github.com/flyteorg/flyteadmin/cmd/main.go:13 +0x91
values-override.yaml
file, it uses this config:
flyteadmin:
additionalVolumes:
- name: cluster-credentials
secret:
secretName: cluster-credentials
additionalVolumeMounts:
- name: cluster-credentials
mountPath: /var/run/credentials
but additionalVolumeMounts
will only mount to the main container. so sync-cluster-resources
isn’t getting the cluster credentials file mounted in the current chart. That’s an easy fix and so now I’m stuck at the PEM error.flyteadmin-token
(starting with data plane in the same cluster before adding a second cluster) and the PEM block error went away.{
"cacert": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----\n",
"token": "..."
}
Ketan (kumare3)
Alex Papanicolaou
02/15/2023, 4:35 AMflytectl
right now due to some mysterious grpc errors:
Error: Connection Info: [Endpoint: dns:///..., InsecureConnection?: false, AuthMode: Pkce]: rpc error: code = Unknown desc = unexpected HTTP status code received from server: 464 (); malformed header: missing HTTP content-type
Error: Connection Info: [Endpoint: dns:///..., InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection closed before server preface received
Not sure what I’m going to do since the docs make it seem easy and it’s just not working. I might try the simple flyte setup and then abandon it if flytectl
still doesn’t work. It worked ~1 month ago but no longer.flytectl
still not working with the simple binary deployment and port-forwarding just like in the deployment docs.
╰─❯ flytectl get projects
Error: Connection Info: [Endpoint: dns:///localhost:8088, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection closed before server preface received
Ketan (kumare3)
Alex Papanicolaou
02/15/2023, 5:10 AM─❯ kubectl -n flyte port-forward service/flyte-backend-flyte-binary 8088:8088 8089:8089
Forwarding from 127.0.0.1:8088 -> 8088
Forwarding from [::1]:8088 -> 8088
Forwarding from 127.0.0.1:8089 -> 8089
Forwarding from [::1]:8089 -> 8089
Handling connection for 8088
flyte-core
deploymentKetan (kumare3)
Alex Papanicolaou
02/15/2023, 5:13 AMKetan (kumare3)
Alex Papanicolaou
02/15/2023, 5:15 AM--insecure
and hadn’t discovered that yet:
╰─❯ flytectl config init --host localhost:8089
This action will overwrite an existing config file at [/Users/alexifm/.flyte/config.yaml]. Do you want to continue? [y/n]: y
Init flytectl config file at [/Users/alexifm/.flyte/config.yaml]%
╰─❯ flytectl get projects
Error: Connection Info: [Endpoint: dns:///localhost:8089, InsecureConnection?: false, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"
Flyte-core is not single binary and port forwarding will jot work for Uiright, I got the ingress working and can visit the console via our DNS name but the connection to GRPC via flytectl wasn’t working
kubefwd
to get flytectl
working since DNS isn’t.
╰─❯ sudo -E kubefwd svc -n flyte -f metadata.name=flyteadmin
╰─❯ cat /etc/hosts
───────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────
│ File: /etc/hosts
───────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────
...
12 │ 127.1.27.1 flyteadmin flyteadmin.flyte flyteadmin-flyte flyteadmin.flyte.svc flyteadmin-flyte-svc flytead
│ min.flyte.svc.cluster.local flyteadmin-flyte-svc-cluster-local flyteadmin.flyte.infima-staging-eks flyteadmin-f
│ lyte-infima-staging-eks flyteadmin.flyte.svc.infima-staging-eks flyteadmin-flyte-svc-infima-staging-eks flytead
│ min.flyte.svc.cluster.infima-staging-eks flyteadmin-flyte-svc-cluster-infima-staging-eks
───────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────
╰─❯ flytectl config init --host flyteadmin:81 --insecure
flytectl
is due to the ingress controller not supporting GRPC. 🤷 We’re going to switch back to the original setup we used from the old version of the docs.Ketan (kumare3)
Yee
Alex Papanicolaou
02/15/2023, 7:59 PM