Jakub Peschel
06/19/2023, 9:02 AM(venv) jpeschel@kinnan:~/Workplace/flyte-demo$ pyflyte run flytedemo.py training_workflow --hyperparameters '{"C": 0.1}'
LogisticRegression(C=0.1, max_iter=3000)
But if I start new local cluster:
(venv) jpeschel@kinnan:~/Workplace/flyte-demo$ ./bin/flytectl demo start
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags.
๐งโ๐ญ Bootstrapping a brand new flyte cluster... ๐จ ๐ง
delete existing sandbox cluster [y/n]:
y
๐ Going to use Flyte v1.7.0 release with image <http://cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-1ae254f8683699b68ecddc89d775fc5d39cc3d84|cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-1ae254f8683699b68ecddc89d775fc5d39cc3d84>
๐ pulling docker image for release <http://cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-1ae254f8683699b68ecddc89d775fc5d39cc3d84|cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-1ae254f8683699b68ecddc89d775fc5d39cc3d84>
๐งโ๐ญ booting Flyte-sandbox container
Waiting for cluster to come up...
Waiting for cluster to come up...
Waiting for cluster to come up...
Waiting for cluster to come up...
Waiting for cluster to come up...
Waiting for cluster to come up...
context flyte-sandbox already exist. Overwriting it
context modified for "flyte-sandbox" and switched over to it.
+-----------------------------------+---------------+-----------+
| SERVICE | STATUS | NAMESPACE |
+-----------------------------------+---------------+-----------+
| k8s: This might take a little bit | Bootstrapping | |
+-----------------------------------+---------------+-----------+
+-----------------------------------+---------------+-----------+
| SERVICE | STATUS | NAMESPACE |
+-----------------------------------+---------------+-----------+
I don't get the expected output from Getting started page and when I try to send the workflow on the cluster I get this error:
(venv) jpeschel@kinnan:~/Workplace/flyte-demo$ pyflyte run --remote flytedemo.py training_workflow --hyperparameters '{"C": 0.1}'
Failed with Exception Code: SYSTEM:Unknown
RPC Failed, with Status: StatusCode.UNAVAILABLE
details: failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127.0.0.1:30080: Socket closed
Debug string UNKNOWN:failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127.0.0.1:30080: Socket closed {created_time:"2023-06-19T09:17:08.725217881+02:00", grpc_status:14}
I tried to check whether the problem is caused by closed ports but netstat showed that port is opened:
(venv) jpeschel@kinnan:~/Workplace/flyte-demo$ sudo netstat -ntlp
[sudo] password for jpeschel:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:6443 0.0.0.0:* LISTEN 101329/docker-proxy
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 3430/systemd-resolv
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 4241/cupsd
tcp 0 0 0.0.0.0:30080 0.0.0.0:* LISTEN 101277/docker-proxy
tcp 0 0 0.0.0.0:30001 0.0.0.0:* LISTEN 101302/docker-proxy
tcp 0 0 0.0.0.0:30000 0.0.0.0:* LISTEN 101316/docker-proxy
tcp 0 0 0.0.0.0:30002 0.0.0.0:* LISTEN 101290/docker-proxy
tcp6 0 0 ::1:631 :::* LISTEN 4241/cupsd
tcp6 0 0 127.0.0.1:63342 :::* LISTEN 15418/java
As well as ping from nmap:
(venv) jpeschel@kinnan:~/Workplace/flyte-demo$ nmap -p 30080 127.0.0.1
Starting Nmap 7.80 ( <https://nmap.org> ) at 2023-06-19 10:06 CEST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000080s latency).
PORT STATE SERVICE
30080/tcp open unknown
I am at the ubuntu 22.0.4.2 LTS, I have 11th Gen Intelยฎ Coreโข i7-11850H @ 2.50GHz ร 16 and 32GiB of memory, which should be more than sufficient for this demo.
Is there something that I didn't do that is required?Samhita Alla
export FLYTECTL_CONFIG=~/.flyte/config-sandbox.yaml
Please make sure to check if config-sandbox.yaml
has the required configuration.Jakub Peschel
06/19/2023, 10:02 AMadmin:
# For GRPC endpoints you might want to use dns:///flyte.myexample.com
endpoint: localhost:30080
authType: Pkce
insecure: true
console:
endpoint: <http://localhost:30080>
logger:
show-source: true
level: 0
Samhita Alla
flytectl demo teardown
flytectl demo start
Jakub Peschel
06/19/2023, 10:07 AM(venv) jpeschel@kinnan:~/Workplace/flyte-demo$ ./bin/flytectl demo teardown
Config cleanup failed. Which Failed due to unlinkat /home/jpeschel/.flyte/k3s/k3s.yaml: permission denied
context removed for "flyte-sandbox".
๐งน ๐งน Sandbox cluster is removed successfully.
โ๏ธ Run the following command to unset sandbox environment variables for accessing flytectl
unset FLYTECTL_CONFIG
(venv) jpeschel@kinnan:~/Workplace/flyte-demo$ ./bin/flytectl demo start
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags.
๐งโ๐ญ Bootstrapping a brand new flyte cluster... ๐จ ๐ง
๐ Going to use Flyte v1.7.0 release with image <http://cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-1ae254f8683699b68ecddc89d775fc5d39cc3d84|cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-1ae254f8683699b68ecddc89d775fc5d39cc3d84>
๐ pulling docker image for release <http://cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-1ae254f8683699b68ecddc89d775fc5d39cc3d84|cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-1ae254f8683699b68ecddc89d775fc5d39cc3d84>
๐งโ๐ญ booting Flyte-sandbox container
Waiting for cluster to come up...
Waiting for cluster to come up...
Waiting for cluster to come up...
Waiting for cluster to come up...
Waiting for cluster to come up...
context modified for "flyte-sandbox" and switched over to it.
Is that the config-sandbox.yaml file?Samhita Alla
Jakub Peschel
06/19/2023, 10:28 AMSamhita Alla
kubectl get pods -n flyte
command and check if all the pods are running?Jakub Peschel
06/19/2023, 10:45 AM(venv) jpeschel@kinnan:~/Workplace/flyte-demo$ kubectl get pods -n flyte
E0619 12:44:09.550062 280188 memcache.go:287] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0619 12:44:09.552732 280188 memcache.go:121] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0619 12:44:09.554045 280188 memcache.go:121] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0619 12:44:09.555097 280188 memcache.go:121] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
NAME READY STATUS RESTARTS AGE
flyte-sandbox-postgresql-0 0/1 Pending 0 29s
flyte-sandbox-proxy-d95874857-xv2xc 0/1 Pending 0 29s
flyte-sandbox-76d484c4b9-p7s2q 0/2 Pending 0 29s
flyte-sandbox-docker-registry-78fb6fd969-qfhq6 0/1 Pending 0 29s
flyte-sandbox-kubernetes-dashboard-6757db879c-slqst 0/1 Pending 0 29s
flyte-sandbox-minio-645c8ddf7c-x6zdn 0/1 Pending 0 29s
Samhita Alla
kubectl logs <pod-name> -n flyte
Jakub Peschel
06/19/2023, 11:10 AM(venv) jpeschel@kinnan:~/Workplace/flyte-demo$ kubectl logs flyte-sandbox-postgresql-0 -n flyte
E0619 13:09:30.664663 305986 memcache.go:287] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0619 13:09:30.670736 305986 memcache.go:121] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0619 13:09:30.672176 305986 memcache.go:121] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
Defaulted container "postgresql" out of: postgresql, init-chmod-data (init)
Samhita Alla
Jakub Peschel
06/19/2023, 1:34 PMjeev