https://flyte.org logo
#ask-the-community
Title
# ask-the-community
s

Stephen Fromm

05/25/2022, 3:11 PM
I can't seem to get this example in the User's Guide working: Flyte Basics: Tasks
I tried using
sandbox
and the
demo
, but I can't make my way through the example with either of them.
Copy code
cd flytesnacks/cookbook/
flytectl sandbox start --source .
k

Ketan (kumare3)

05/25/2022, 3:15 PM
Wow that is basic
Can you just use pyflyte run
s

Stephen Fromm

05/25/2022, 3:16 PM
I've gotten
pyflyte run
to work in other examples; this one I'm trying to use
flytectl
k

Ketan (kumare3)

05/25/2022, 3:17 PM
Ohh
You want to use package
s

Stephen Fromm

05/25/2022, 3:18 PM
Just a minute...let me see if setting some environment variables makes a difference
OK, now I'm running
Copy code
flytectl sandbox exec -- docker build . --tag "flytebasics:v1" -f core/Dockerfile
That will take a few minutes. But it's the step I've had problems with before. (Meaning, it runs, but things after that don't work.)
After that step, the console (UI) is no longer there.
Copy code
pyflyte --pkgs core.flyte_basics package --image flytebasics:v1
runs fine
But
Copy code
flytectl register files --project flytesnacks --domain development --archive flyte-package.tgz --version v1
results in
Copy code
Error: rpc error: code = Unavailable desc = connection closed
k

Ketan (kumare3)

05/25/2022, 3:54 PM
Btw @Stephen Fromm this is getting simplified in the next release of flytekit - we have pyflyte register, But let's try and solve this. Cc @Prafulla Mahindrakar can you help
👍 2
p

Prafulla Mahindrakar

05/25/2022, 3:56 PM
hi @Stephen Fromm. what the contents of config.yaml file and you mentioned the UI is also not accessible?
you can run simple command to get debug level logs too . Communication issues with the admin .
Copy code
flytectl get projects --logger.level=6
s

Stephen Fromm

05/25/2022, 3:57 PM
After I started the sandbox, I ran
Copy code
export KUBECONFIG=$KUBECONFIG:/Users/sfromm/.kube/config:/Users/sfromm/.flyte/k3s/k3s.yaml
export FLYTECTL_CONFIG=/Users/sfromm/.flyte/config-sandbox.yaml
p

Prafulla Mahindrakar

05/25/2022, 3:57 PM
Also can you check the status of your flyte pods .
Copy code
kubectl get pods -n flyte
s

Stephen Fromm

05/25/2022, 3:58 PM
`config-sandbox.yaml`:
Copy code
admin:
 # For GRPC endpoints you might want to use dns:///flyte.myexample.com
 endpoint: localhost:30081
 authType: Pkce
 insecure: true
logger:
 show-source: true
 level: 0
p

Prafulla Mahindrakar

05/25/2022, 3:58 PM
That seems right. Also can you check the status of your pods . Did you use demo or sandbox ?
s

Stephen Fromm

05/25/2022, 3:59 PM
I used sandbox. In the past I tried demo but that didn't work either.
p

Prafulla Mahindrakar

05/25/2022, 3:59 PM
Ok in the current state can you run
kubectl get pods -n flyte
s

Stephen Fromm

05/25/2022, 3:59 PM
status of pods...none are ready
Copy code
NAME                     READY  STATUS   RESTARTS  AGE
syncresources-27558199-7t75r         0/1   Completed  0     40m
syncresources-27558200-kc9l7         0/1   Completed  0     39m
postgres-575f67c7f8-wj5bf           0/1   Evicted   0     41m
postgres-575f67c7f8-h9br5           0/1   Evicted   0     39m
postgres-575f67c7f8-2dxzc           0/1   Evicted   0     39m
postgres-575f67c7f8-wlvqr           0/1   Evicted   0     39m
postgres-575f67c7f8-2pz4x           0/1   Evicted   0     39m
postgres-575f67c7f8-2gvqq           0/1   Evicted   0     39m
postgres-575f67c7f8-nmj6b           0/1   Evicted   0     39m
postgres-575f67c7f8-cshj8           0/1   Evicted   0     39m
postgres-575f67c7f8-jhvxk           0/1   Evicted   0     39m
postgres-575f67c7f8-hd8dt           0/1   Evicted   0     39m
postgres-575f67c7f8-pg5tg           0/1   Evicted   0     39m
postgres-575f67c7f8-dwsgk           0/1   Evicted   0     38m
postgres-575f67c7f8-gtlj9           0/1   Pending   0     38m
minio-799fc77896-kfspb            0/1   Evicted   0     41m
minio-799fc77896-g5fq4            0/1   Pending   0     38m
redoc-54598f6688-98prm            0/1   Evicted   0     41m
redoc-54598f6688-r68x4            0/1   Pending   0     38m
syncresources-27558201-8489k         0/1   Pending   0     38m
flyte-pod-webhook-69c894c9d-489hq       0/1   Evicted   0     41m
flyte-pod-webhook-69c894c9d-vg7bq       0/1   Pending   0     38m
flyte-contour-contour-55bf854cdf-58rdj    0/1   Evicted   0     41m
flyte-contour-contour-55bf854cdf-rstss    0/1   Pending   0     38m
flyte-kubernetes-dashboard-7fd989b99d-r9hmt  0/1   Evicted   0     41m
flyte-kubernetes-dashboard-7fd989b99d-jbkw4  0/1   Pending   0     38m
flyteconsole-585d84889b-crmtd         0/1   Evicted   0     41m
flyteconsole-585d84889b-4vqsq         0/1   Pending   0     38m
flytepropeller-559f9ddb6f-nnzw5        0/1   Evicted   0     41m
flytepropeller-559f9ddb6f-rk5ss        0/1   Pending   0     38m
flyteadmin-6d4f5c7c77-4z4d4          0/1   Evicted   0     41m
flyteadmin-6d4f5c7c77-4qjm6          0/1   Pending   0     38m
flytescheduler-66db59bc95-sk27k        0/1   Evicted   0     41m
flytescheduler-66db59bc95-5r2sh        0/1   Pending   0     38m
datacatalog-8478745fc4-fp9lg         0/1   Evicted   0     41m
datacatalog-8478745fc4-6klb5         0/1   Pending   0     37m
syncresources-27558202-bw6fr         0/1   Pending   0     37m
syncresources-27558203-mbt5h         0/1   Pending   0     36m
syncresources-27558204-v6j5q         0/1   Pending   0     35m
syncresources-27558205-d2nnx         0/1   Pending   0     34m
syncresources-27558206-88kfh         0/1   Pending   0     33m
syncresources-27558207-7flx4         0/1   Pending   0     32m
syncresources-27558208-jh4m6         0/1   Pending   0     31m
syncresources-27558209-7vqlp         0/1   Pending   0     30m
syncresources-27558210-wzdlm         0/1   Pending   0     29m
syncresources-27558211-wckvh         0/1   Pending   0     28m
syncresources-27558212-bzkjn         0/1   Pending   0     27m
syncresources-27558213-jdrhm         0/1   Pending   0     26m
syncresources-27558214-p5wb8         0/1   Pending   0     25m
syncresources-27558215-bnh4l         0/1   Pending   0     24m
syncresources-27558216-sls7q         0/1   Pending   0     23m
syncresources-27558217-psq69         0/1   Pending   0     22m
syncresources-27558218-mxgbj         0/1   Pending   0     21m
syncresources-27558219-cksbf         0/1   Pending   0     20m
syncresources-27558220-5459k         0/1   Pending   0     19m
syncresources-27558221-24fmr         0/1   Pending   0     18m
syncresources-27558222-wtnwq         0/1   Pending   0     17m
syncresources-27558223-xsxqs         0/1   Pending   0     16m
syncresources-27558224-t2wg8         0/1   Pending   0     15m
syncresources-27558225-8c62x         0/1   Pending   0     14m
syncresources-27558226-rxjxt         0/1   Pending   0     13m
syncresources-27558227-8hjph         0/1   Pending   0     12m
syncresources-27558228-p9h4l         0/1   Pending   0     11m
syncresources-27558229-dglks         0/1   Pending   0     10m
syncresources-27558230-6c9zr         0/1   Pending   0     9m53s
syncresources-27558231-w5ddr         0/1   Pending   0     8m53s
syncresources-27558232-lznxd         0/1   Pending   0     7m53s
syncresources-27558233-q9kk6         0/1   Pending   0     6m53s
flyte-contour-envoy-z8zlg           0/2   Evicted   0     6m45s
syncresources-27558234-4dd82         0/1   Pending   0     5m53s
syncresources-27558235-lls2m         0/1   Pending   0     4m53s
syncresources-27558236-kpqx6         0/1   Pending   0     3m53s
syncresources-27558237-cgrdf         0/1   Pending   0     2m53s
syncresources-27558238-drjtx         0/1   Pending   0     113s
syncresources-27558239-jbg2x         0/1   Pending   0     53s
The UI console is there until I run the docker build command. Then the console is unavailable.
p

Prafulla Mahindrakar

05/25/2022, 4:01 PM
This might indicate some scheduling issues due to resource crunch. can you check postgres pod why its in pending , by doing describe
s

Stephen Fromm

05/25/2022, 4:01 PM
Yeah
What's the full command? (I'm very new to k8s)
p

Prafulla Mahindrakar

05/25/2022, 4:05 PM
Copy code
kubectl describe pod -n flyte postgres-575f67c7f8-gtlj9
s

Stephen Fromm

05/25/2022, 4:06 PM
Copy code
Do you want the full output?  Partial output:
Events:
 Type   Reason      Age         From        Message
 ----   ------      ----        ----        -------
 Warning FailedScheduling 45m         default-scheduler 0/1 nodes are available: 1 node(s) had taint {<http://node.kubernetes.io/disk-pressure|node.kubernetes.io/disk-pressure>: }, that the pod didn't tolerate.
 Warning FailedScheduling 19m (x31 over 45m) default-scheduler 0/1 nodes are available: 1 node(s) had taint {<http://node.kubernetes.io/disk-pressure|node.kubernetes.io/disk-pressure>: }, that the pod didn't tolerate.
p

Prafulla Mahindrakar

05/25/2022, 4:07 PM
This indicates you node is having space issues. You can do
Copy code
docker system prune -a --volumes
s

Stephen Fromm

05/25/2022, 4:08 PM
OK, after that completes, should I just try again?
p

Prafulla Mahindrakar

05/25/2022, 4:08 PM
yes
Do a teardown first
👍 1
Copy code
flytectl sandbox teardown
s

Stephen Fromm

05/25/2022, 4:09 PM
OK, I'll try all that. If I continue to have problems I'll post in this thread.
👍 1
p

Prafulla Mahindrakar

05/25/2022, 4:10 PM
Also if you run into same space issues , would suggest increase node disk size. If you are using docker desktop you can increase it from its resource setting in there console
👍 1
s

Stephen Fromm

05/25/2022, 4:25 PM
Excellent! That worked. (Didn't have to increase node disk size.). Thanks so much for the help!
k

Ketan (kumare3)

05/25/2022, 4:28 PM
This is in trouble shooting
s

Stephen Fromm

05/25/2022, 4:29 PM
You mean, in the docs?
k

Ketan (kumare3)

05/25/2022, 4:30 PM
Ya
Can you help add it in the right place
s

Stephen Fromm

05/25/2022, 4:30 PM
Via a pull request or something?
k

Ketan (kumare3)

05/25/2022, 4:51 PM
ya if you can that would be helpful 🙂
103 Views