https://flyte.org
Join Slack
I can't seem to get this example in the User's Guide working: <Flyte Basics: Tasks>
# flyte-support
f
I can't seem to get this example in the User's Guide working: Flyte Basics: Tasks
I tried using 
sandbox
and the 
demo
, but I can't make my way through the example with either of them.
Copy code
cd flytesnacks/cookbook/
flytectl sandbox start --source .
f
Wow that is basic
Can you just use pyflyte run
f
I've gotten 
pyflyte run
to work in other examples; this one I'm trying to use 
flytectl
f
Ohh
You want to use package
f
Just a minute...let me see if setting some environment variables makes a difference
OK, now I'm running
Copy code
flytectl sandbox exec -- docker build . --tag "flytebasics:v1" -f core/Dockerfile
That will take a few minutes. But it's the step I've had problems with before. (Meaning, it runs, but things after that don't work.)
After that step, the console (UI) is no longer there.
Copy code
pyflyte --pkgs core.flyte_basics package --image flytebasics:v1
runs fine
But
Copy code
flytectl register files --project flytesnacks --domain development --archive flyte-package.tgz --version v1
results in
Copy code
Error: rpc error: code = Unavailable desc = connection closed
f
Btw @fancy-leather-59976 this is getting simplified in the next release of flytekit - we have pyflyte register, But let's try and solve this. Cc @icy-agent-73298 can you help
👍 2
i
hi @fancy-leather-59976. what the contents of config.yaml file and you mentioned the UI is also not accessible?
you can run simple command to get debug level logs too . Communication issues with the admin .
Copy code
flytectl get projects --logger.level=6
f
After I started the sandbox, I ran
Copy code
export KUBECONFIG=$KUBECONFIG:/Users/sfromm/.kube/config:/Users/sfromm/.flyte/k3s/k3s.yaml
export FLYTECTL_CONFIG=/Users/sfromm/.flyte/config-sandbox.yaml
i
Also can you check the status of your flyte pods .
Copy code
kubectl get pods -n flyte
f
`config-sandbox.yaml`:
Copy code
admin:
 # For GRPC endpoints you might want to use dns:///flyte.myexample.com
 endpoint: localhost:30081
 authType: Pkce
 insecure: true
logger:
 show-source: true
 level: 0
i
That seems right. Also can you check the status of your pods . Did you use demo or sandbox ?
f
I used sandbox. In the past I tried demo but that didn't work either.
i
Ok in the current state can you run 
kubectl get pods -n flyte
f
status of pods...none are ready
Copy code
NAME                     READY  STATUS   RESTARTS  AGE
syncresources-27558199-7t75r         0/1   Completed  0     40m
syncresources-27558200-kc9l7         0/1   Completed  0     39m
postgres-575f67c7f8-wj5bf           0/1   Evicted   0     41m
postgres-575f67c7f8-h9br5           0/1   Evicted   0     39m
postgres-575f67c7f8-2dxzc           0/1   Evicted   0     39m
postgres-575f67c7f8-wlvqr           0/1   Evicted   0     39m
postgres-575f67c7f8-2pz4x           0/1   Evicted   0     39m
postgres-575f67c7f8-2gvqq           0/1   Evicted   0     39m
postgres-575f67c7f8-nmj6b           0/1   Evicted   0     39m
postgres-575f67c7f8-cshj8           0/1   Evicted   0     39m
postgres-575f67c7f8-jhvxk           0/1   Evicted   0     39m
postgres-575f67c7f8-hd8dt           0/1   Evicted   0     39m
postgres-575f67c7f8-pg5tg           0/1   Evicted   0     39m
postgres-575f67c7f8-dwsgk           0/1   Evicted   0     38m
postgres-575f67c7f8-gtlj9           0/1   Pending   0     38m
minio-799fc77896-kfspb            0/1   Evicted   0     41m
minio-799fc77896-g5fq4            0/1   Pending   0     38m
redoc-54598f6688-98prm            0/1   Evicted   0     41m
redoc-54598f6688-r68x4            0/1   Pending   0     38m
syncresources-27558201-8489k         0/1   Pending   0     38m
flyte-pod-webhook-69c894c9d-489hq       0/1   Evicted   0     41m
flyte-pod-webhook-69c894c9d-vg7bq       0/1   Pending   0     38m
flyte-contour-contour-55bf854cdf-58rdj    0/1   Evicted   0     41m
flyte-contour-contour-55bf854cdf-rstss    0/1   Pending   0     38m
flyte-kubernetes-dashboard-7fd989b99d-r9hmt  0/1   Evicted   0     41m
flyte-kubernetes-dashboard-7fd989b99d-jbkw4  0/1   Pending   0     38m
flyteconsole-585d84889b-crmtd         0/1   Evicted   0     41m
flyteconsole-585d84889b-4vqsq         0/1   Pending   0     38m
flytepropeller-559f9ddb6f-nnzw5        0/1   Evicted   0     41m
flytepropeller-559f9ddb6f-rk5ss        0/1   Pending   0     38m
flyteadmin-6d4f5c7c77-4z4d4          0/1   Evicted   0     41m
flyteadmin-6d4f5c7c77-4qjm6          0/1   Pending   0     38m
flytescheduler-66db59bc95-sk27k        0/1   Evicted   0     41m
flytescheduler-66db59bc95-5r2sh        0/1   Pending   0     38m
datacatalog-8478745fc4-fp9lg         0/1   Evicted   0     41m
datacatalog-8478745fc4-6klb5         0/1   Pending   0     37m
syncresources-27558202-bw6fr         0/1   Pending   0     37m
syncresources-27558203-mbt5h         0/1   Pending   0     36m
syncresources-27558204-v6j5q         0/1   Pending   0     35m
syncresources-27558205-d2nnx         0/1   Pending   0     34m
syncresources-27558206-88kfh         0/1   Pending   0     33m
syncresources-27558207-7flx4         0/1   Pending   0     32m
syncresources-27558208-jh4m6         0/1   Pending   0     31m
syncresources-27558209-7vqlp         0/1   Pending   0     30m
syncresources-27558210-wzdlm         0/1   Pending   0     29m
syncresources-27558211-wckvh         0/1   Pending   0     28m
syncresources-27558212-bzkjn         0/1   Pending   0     27m
syncresources-27558213-jdrhm         0/1   Pending   0     26m
syncresources-27558214-p5wb8         0/1   Pending   0     25m
syncresources-27558215-bnh4l         0/1   Pending   0     24m
syncresources-27558216-sls7q         0/1   Pending   0     23m
syncresources-27558217-psq69         0/1   Pending   0     22m
syncresources-27558218-mxgbj         0/1   Pending   0     21m
syncresources-27558219-cksbf         0/1   Pending   0     20m
syncresources-27558220-5459k         0/1   Pending   0     19m
syncresources-27558221-24fmr         0/1   Pending   0     18m
syncresources-27558222-wtnwq         0/1   Pending   0     17m
syncresources-27558223-xsxqs         0/1   Pending   0     16m
syncresources-27558224-t2wg8         0/1   Pending   0     15m
syncresources-27558225-8c62x         0/1   Pending   0     14m
syncresources-27558226-rxjxt         0/1   Pending   0     13m
syncresources-27558227-8hjph         0/1   Pending   0     12m
syncresources-27558228-p9h4l         0/1   Pending   0     11m
syncresources-27558229-dglks         0/1   Pending   0     10m
syncresources-27558230-6c9zr         0/1   Pending   0     9m53s
syncresources-27558231-w5ddr         0/1   Pending   0     8m53s
syncresources-27558232-lznxd         0/1   Pending   0     7m53s
syncresources-27558233-q9kk6         0/1   Pending   0     6m53s
flyte-contour-envoy-z8zlg           0/2   Evicted   0     6m45s
syncresources-27558234-4dd82         0/1   Pending   0     5m53s
syncresources-27558235-lls2m         0/1   Pending   0     4m53s
syncresources-27558236-kpqx6         0/1   Pending   0     3m53s
syncresources-27558237-cgrdf         0/1   Pending   0     2m53s
syncresources-27558238-drjtx         0/1   Pending   0     113s
syncresources-27558239-jbg2x         0/1   Pending   0     53s
The UI console is there until I run the docker build command. Then the console is unavailable.
i
This might indicate some scheduling issues due to resource crunch. can you check postgres pod why its in pending , by doing describe
f
Yeah
What's the full command? (I'm very new to k8s)
i
Copy code
kubectl describe pod -n flyte postgres-575f67c7f8-gtlj9
f
Copy code
Do you want the full output?  Partial output:
Events:
 Type   Reason      Age         From        Message
 ----   ------      ----        ----        -------
 Warning FailedScheduling 45m         default-scheduler 0/1 nodes are available: 1 node(s) had taint {<http://node.kubernetes.io/disk-pressure|node.kubernetes.io/disk-pressure>: }, that the pod didn't tolerate.
 Warning FailedScheduling 19m (x31 over 45m) default-scheduler 0/1 nodes are available: 1 node(s) had taint {<http://node.kubernetes.io/disk-pressure|node.kubernetes.io/disk-pressure>: }, that the pod didn't tolerate.
i
This indicates you node is having space issues. You can do
Copy code
docker system prune -a --volumes
f
OK, after that completes, should I just try again?
i
yes
Do a teardown first
👍 1
Copy code
flytectl sandbox teardown
f
OK, I'll try all that. If I continue to have problems I'll post in this thread.
👍 1
i
Also if you run into same space issues , would suggest increase node disk size. If you are using docker desktop you can increase it from its resource setting in there console
👍 1
f
Excellent! That worked. (Didn't have to increase node disk size.). Thanks so much for the help!
f
This is in trouble shooting
f
You mean, in the docs?
f
Ya
Can you help add it in the right place
f
Via a pull request or something?
f
ya if you can that would be helpful 🙂
198 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by