Just to make sure I m not doing this very inefficiently If I

Question

Just to make sure I m not doing this very inefficiently If I m just testing flyte locally and developing a workflow the fastest way is to run `flytectl demo start` and for docker images I should run something like this ```do docker build f docker Dockerfile $x t $x docker tag $x localhost 30000 $x latest docker push localhost 30000 $x latest done``` There is no way to utilize the local docker images correct my images are 5gb + so docker push is very slow which is why i m asking

Dan Farrell · Answer

Also Does anyone know why the docker registry started by `flytectl demo start` might be dying ```kubectl get pods n flyte NAME READY STATUS RESTARTS AGE flyte sandbox docker registry 759844bc88 4pfrz 0 1 Error 0 79m flyte sandbox docker registry 759844bc88 gbn5q 0 1 Evicted 0 4m46s flyte sandbox docker registry 759844bc88 cgjls 0 1 Evicted 0 4m46s flyte sandbox docker registry 759844bc88 trxmm 0 1 Evicted 0 4m46s flyte sandbox docker registry 759844bc88 2t8jk 0 1 Evicted 0 4m46s flyte sandbox docker registry 759844bc88 6s5hr 0 1 Evicted 0 4m46s flyte sandbox docker registry 759844bc88 p7bbl 0 1 Evicted 0 4m46s flyte sandbox docker registry 759844bc88 bm8gh 0 1 Evicted 0 4m45s flyte sandbox docker registry 759844bc88 wf47p 0 1 Evicted 0 4m45s flyte sandbox docker registry 759844bc88 pk2jk 0 1 Evicted 0 4m44s flyte sandbox docker registry 759844bc88 8cj8q 0 1 Evicted 0 4m44s flyte sandbox docker registry 759844bc88 56jhj 0 1 Evicted 0 4m43s flyte sandbox docker registry 759844bc88 bhq5g 0 1 Pending 0 4m43s flyte sandbox 7d699df5fc z9rgf 0 1 Error 0 79m flyte sandbox 7d699df5fc xsd2x 0 1 Pending 0 4m14s flyte sandbox postgresql 0 0 1 Pending 0 4m12s flyte sandbox kubernetes dashboard 6757db879c jmwrl 0 1 Error 0 79m flyte sandbox kubernetes dashboard 6757db879c h2d74 0 1 Pending 0 4m11s flyte sandbox proxy d95874857 m8vbc 0 1 ContainerStatusUnknown 1 79m flyte sandbox proxy d95874857 j7h8v 0 1 Pending 0 4m9s flyte sandbox minio 645c8ddf7c bkxhx 0 1 ContainerStatusUnknown 1 79m flyte sandbox minio 645c8ddf7c kldth 0 1 Pending 0 4m7s flyte sandbox buildkit 7d7d55dbb 8lrs4 0 1 Error 0 79m flyte sandbox buildkit 7d7d55dbb 4kk47 0 1 Pending 0 4m5s```

Dan Farrell · Answer

There s some logs in the master kube container ```E1011 13 24 02 242497 86 kubelet go 1294 Image garbage collection failed multiple times in a row err= failed to garbage collect required amount of images Wanted to free 25381440716 bytes but freed 0 bytes W1011 13 24 02 245951 86 info go 53 Couldn t collect info from any of the files in etc machine id var lib dbus machine id I1011 13 29 02 243396 86 image gc manager go 310 Disk usage on image filesystem is over the high threshold trying to free bytes down to the low threshold usage=92 highThreshold=85 amountToFree=26957929676 lo wThreshold=80 W1011 13 29 02 246713 86 info go 53 Couldn t collect info from any of the files in etc machine id var lib dbus machine id E1011 13 29 02 247249 86 kubelet go 1294 Image garbage collection failed multiple times in a row err= failed to garbage collect required amount of images Wanted to free 26957929676 bytes but freed 0 bytes W1011 13 34 02 246709 86 info go 53 Couldn t collect info from any of the files in etc machine id var lib dbus machine id I1011 13 34 02 248323 86 image gc manager go 310 Disk usage on image filesystem is over the high threshold trying to free bytes down to the low threshold usage=92 highThreshold=85 amountToFree=28551834828 lo wThreshold=80```

Dan Farrell · Answer

seems like k3s doesn t think 30 gbs is enough

Samhita Alla · Answer

Um haven t across this error before Have you tried tearing down your cluster and restarting it