https://flyte.org logo
Join Slack
Powered by
Just to make sure I'm not doing this very ineffici...
# flyte-support
s
Just to make sure I'm not doing this very inefficiently... If I'm just testing flyte locally and developing a workflow, the fastest way is to run 
flytectl demo start
and for docker images I should run something like this:
Copy code
do
  docker build -f docker/Dockerfile.$x -t $x .
  docker tag $x localhost:30000/$x:latest
  docker push localhost:30000/$x:latest
done
There is no way to utilize the local docker images... correct? (my images are 5gb + so docker push is very slow - which is why i'm asking)
Also, Does anyone know why the docker-registry started by 
flytectl demo start
might be dying?
Copy code
kubectl get pods -n flyte
NAME                                                  READY   STATUS                   RESTARTS   AGE
flyte-sandbox-docker-registry-759844bc88-4pfrz        0/1     Error                    0          79m
flyte-sandbox-docker-registry-759844bc88-gbn5q        0/1     Evicted                  0          4m46s
flyte-sandbox-docker-registry-759844bc88-cgjls        0/1     Evicted                  0          4m46s
flyte-sandbox-docker-registry-759844bc88-trxmm        0/1     Evicted                  0          4m46s
flyte-sandbox-docker-registry-759844bc88-2t8jk        0/1     Evicted                  0          4m46s
flyte-sandbox-docker-registry-759844bc88-6s5hr        0/1     Evicted                  0          4m46s
flyte-sandbox-docker-registry-759844bc88-p7bbl        0/1     Evicted                  0          4m46s
flyte-sandbox-docker-registry-759844bc88-bm8gh        0/1     Evicted                  0          4m45s
flyte-sandbox-docker-registry-759844bc88-wf47p        0/1     Evicted                  0          4m45s
flyte-sandbox-docker-registry-759844bc88-pk2jk        0/1     Evicted                  0          4m44s
flyte-sandbox-docker-registry-759844bc88-8cj8q        0/1     Evicted                  0          4m44s
flyte-sandbox-docker-registry-759844bc88-56jhj        0/1     Evicted                  0          4m43s
flyte-sandbox-docker-registry-759844bc88-bhq5g        0/1     Pending                  0          4m43s
flyte-sandbox-7d699df5fc-z9rgf                        0/1     Error                    0          79m
flyte-sandbox-7d699df5fc-xsd2x                        0/1     Pending                  0          4m14s
flyte-sandbox-postgresql-0                            0/1     Pending                  0          4m12s
flyte-sandbox-kubernetes-dashboard-6757db879c-jmwrl   0/1     Error                    0          79m
flyte-sandbox-kubernetes-dashboard-6757db879c-h2d74   0/1     Pending                  0          4m11s
flyte-sandbox-proxy-d95874857-m8vbc                   0/1     ContainerStatusUnknown   1          79m
flyte-sandbox-proxy-d95874857-j7h8v                   0/1     Pending                  0          4m9s
flyte-sandbox-minio-645c8ddf7c-bkxhx                  0/1     ContainerStatusUnknown   1          79m
flyte-sandbox-minio-645c8ddf7c-kldth                  0/1     Pending                  0          4m7s
flyte-sandbox-buildkit-7d7d55dbb-8lrs4                0/1     Error                    0          79m
flyte-sandbox-buildkit-7d7d55dbb-4kk47                0/1     Pending                  0          4m5s
There's some logs in the master kube container:
Copy code
E1011 13:24:02.242497      86 kubelet.go:1294] "Image garbage collection failed multiple times in a row" err="failed to garbage collect required amount of images. Wanted to free 25381440716 bytes, but freed 0 bytes" 
W1011 13:24:02.245951      86 info.go:53] Couldn't collect info from any of the files in "/etc/machine-id,/var/lib/dbus/machine-id"                                                                                     
I1011 13:29:02.243396      86 image_gc_manager.go:310] "Disk usage on image filesystem is over the high threshold, trying to free bytes down to the low threshold" usage=92 highThreshold=85 amountToFree=26957929676 lo
wThreshold=80                                                                                                                                                                                                           
W1011 13:29:02.246713      86 info.go:53] Couldn't collect info from any of the files in "/etc/machine-id,/var/lib/dbus/machine-id"                                                                                     
E1011 13:29:02.247249      86 kubelet.go:1294] "Image garbage collection failed multiple times in a row" err="failed to garbage collect required amount of images. Wanted to free 26957929676 bytes, but freed 0 bytes" 
W1011 13:34:02.246709      86 info.go:53] Couldn't collect info from any of the files in "/etc/machine-id,/var/lib/dbus/machine-id"                                                                                     
I1011 13:34:02.248323      86 image_gc_manager.go:310] "Disk usage on image filesystem is over the high threshold, trying to free bytes down to the low threshold" usage=92 highThreshold=85 amountToFree=28551834828 lo
wThreshold=80
seems like k3s doesn't think 30 gbs is enough.....
t
Um haven't across this error before. Have you tried tearing down your cluster and restarting it?
10 Views
Open in Slack
PreviousNext
Powered by