Hi I get those errors on my Mac M1 when following the quicks

Question

Hi I get those errors on my Mac M1 when following the quickstart tutorial Thank you for the tool and the time ` Error syncing pod skipping err= failed to StartContainer for minio with CrashLoopBackOff back off 5m0s restarting failed container=minio pod=flyte sandbox minio 645c8ddf7c rrj5x flyte 9d8637fe a97e 40e7 a5a6 985bc0fbc21b pod= flyte flyte sandbox minio 645c8ddf7c rrj5x podUID=9d8637fe a97e 40e7 a5a6 985bc0fbc21b` `I0519 19 26 18 275183 59 scope go 110 RemoveContainer containerID= 38a7520207d8aaaa55ca89553770b83c086e9d2e671580fbbe85f0b34cc8608a ` `E0519 19 26 18 276055 59 pod workers go 951 Error syncing pod skipping err= failed to StartContainer for flyte with CrashLoopBackOff back off 5m0s restarting failed container=flyte pod=flyte sandbox b789778f6 c6lxd flyte 94b613f8 e323 4cf8 b250 bfcfc1bb1f4a pod= flyte flyte sandbox b789778f6 c6lxd podUID=94b613f8 e323 4cf8 b250 bfcfc1bb1f4a`

Yee · Answer

what was the command you ran to start everything

Harry Souris · Answer

this one flytectl demo start

Harry Souris · Answer

this is the tuttorial i follow <https docs flyte org projects cookbook en latest index html>

Harry Souris · Answer

also have no proxy=localhost 127 0 0 1

Harry Souris · Answer

i see this too

Harry Souris · Answer

```INFO 0000 0 Couldn t find a config file Relying on env vars and pflags ```

Yee · Answer

i don t understand the significance of the no proxy where is that set

Yee · Answer

you mean as env var

Yee · Answer

on host or in the container

Yee · Answer

can you `kubectl n flyte get pod`

Yee · Answer

and `kubectl n flyte describe lt all failing pods gt `

Harry Souris · Answer

well don t know how and why but after deleting everything and pulling a new image i can see the flyte console

Harry Souris · Answer

when executing the example i get an error

Yee · Answer

i see

Yee · Answer

what s the error

Harry Souris · Answer

pyflyte run remote wine flyte py training workflow hyperparameters C 0 1 Failed with Unknown Exception lt class requests exceptions ConnectionError gt Reason Connection aborted RemoteDisconnected Remote end closed connection without response Connection aborted RemoteDisconnected Remote end closed connection without response

Yee · Answer

kubectl get pods

Yee · Answer

n flyte

Yee · Answer

anything failing

Harry Souris · Answer

NAME READY STATUS RESTARTS AGE flyte sandbox docker registry 7744c9999 lbxzc 1 1 Running 0 5m51s flyte sandbox proxy d95874857 r5lzh 1 1 Running 0 5m51s flyte sandbox kubernetes dashboard 6757db879c 6t2rf 1 1 Running 0 5m51s flyte sandbox b789778f6 hw96t 1 1 Running 0 5m51s flyte sandbox postgresql 0 1 1 Running 1 77s ago 5m51s flyte sandbox minio 645c8ddf7c cchgz 0 1 CrashLoopBackOff 5 23s ago 5m51s

Yee · Answer

i see

Harry Souris · Answer

i think my resources in the cluster

Yee · Answer

yeah minio shouldn t be crashing

Yee · Answer

`describe` it

Yee · Answer

and `logs p`

Yee · Answer

it probably doesn t have logs if it s crashing but it might

Harry Souris · Answer

no logs

Harry Souris · Answer

I think i will give more resources to the cluster more space restart it and let you know

Yee · Answer

k

Yee · Answer

describe has more info too often

Yee · Answer

in the events section

Harry Souris · Answer

by the way why i see this INFO 0000 0 Couldn t find a config file Relying on env vars and pflags

Yee · Answer

depends on where you re seeing it

Harry Souris · Answer

even if I have set it up correctly the config file

Yee · Answer

copy paste the full command stacktrace

Harry Souris · Answer

flytectl demo start INFO 0000 0 Couldn t find a config file Relying on env vars and pflags factory worker Bootstrapping a brand new flyte cluster hammer wrench whale2 Going to use Flyte v1 6 0 release with image whale2 pulling docker image for release

Yee · Answer

oh i think that s fine actually

Yee · Answer

shouldn t matter for that command

Yee · Answer

but it s looking for a file at `~ flyte config yaml`

Harry Souris · Answer

waiting to start

Harry Souris · Answer

now i have a problem with the flyte console

Harry Souris · Answer

flyte sandbox proxy d95874857 6hqf2 1 1 Running 0 9m2s flyte sandbox kubernetes dashboard 6757db879c txnwr 1 1 Running 0 9m2s flyte sandbox docker registry 7744c9999 kw497 1 1 Running 0 9m2s flyte sandbox minio 645c8ddf7c q8492 1 1 Running 5 3m44s ago 9m2s flyte sandbox postgresql 0 1 1 Running 2 2m59s ago 9m2s flyte sandbox b789778f6 fwkvq 0 1 CrashLoopBackOff 6 40s ago 9m2s

Yee · Answer

logs

Harry Souris · Answer

json src composite workqueue go 88 level debug msg Subqueue handler batch round ts 2023 05 19T21 53 48Z json src composite workqueue go 98 level debug msg Dynamically configured batch size 1 ts 2023 05 19T21 53 48Z json src composite workqueue go 129 level debug msg Exiting SubQueue handler batch round ts 2023 05 19T21 53 48Z json src composite workqueue go 88 level debug msg Subqueue handler batch round ts 2023 05 19T21 53 49Z json src composite workqueue go 98 level debug msg Dynamically configured batch size 1 ts 2023 05 19T21 53 49Z json src composite workqueue go 129 level debug msg Exiting SubQueue handler batch round ts 2023 05 19T21 53 49Z json src composite workqueue go 88 level debug msg Subqueue handler batch round ts 2023 05 19T21 53 50Z json src composite workqueue go 98 level debug msg Dynamically configured batch size 1 ts 2023 05 19T21 53 50Z json src composite workqueue go 129 level debug msg Exiting SubQueue handler batch round ts 2023 05 19T21 53 50Z json src composite workqueue go 88 level debug msg Subqueue handler batch round ts 2023 05 19T21 53 51Z json src composite workqueue go 98 level debug msg Dynamically configured batch size 1 ts 2023 05 19T21 53 51Z json src composite workqueue go 129 level debug msg Exiting SubQueue handler batch round ts 2023 05 19T21 53 51Z

Yee · Answer

reload the web page

Harry Souris · Answer

kk

Harry Souris · Answer

```upstream request timeout```

Harry Souris · Answer

in the cluster is like this E0519 21 58 50 734034 57 pod workers go 951 Error syncing pod skipping err= failed to StartContainer for flyte with CrashLoopBackOff back off 5m0s restarting failed container=flyte pod=flyte sandbox b789778f6 fwkvq flyte ce5aaf96 ef0e 4985 8d9a 7b1a75d33d73 pod= flyte flyte sandbox b789778f6 fwkvq podUID=ce5aaf96 ef0e 4985 8d9a 7b1a75d33d73

Harry Souris · Answer

the same message as before

Yee · Answer

the pod is still crashing

Yee · Answer

can you describe

Yee · Answer

and copy paste

Harry Souris · Answer

kubectl describe pods Name py39 cacher Namespace default Priority 0 Service Account default Node 856e6b497fcd 172 17 0 2 Start Time Sat 20 May 2023 00 42 55 +0300 Labels lt none gt Annotations lt none gt Status Succeeded IP 10 42 0 11 IPs IP 10 42 0 11 Containers flytekit Container ID c Image Image ID Port lt none gt Host Port lt none gt Command echo Args Flyte State Terminated Reason Completed Exit Code 0 Started Sat 20 May 2023 00 47 43 +0300 Finished Sat 20 May 2023 00 47 43 +0300 Ready False Restart Count 0 Environment lt none gt Mounts var run secrets kubernetes io serviceaccount from kube api access mxptd ro Conditions Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes kube api access mxptd Type Projected a volume that contains injected data from multiple sources TokenExpirationSeconds 3607 ConfigMapName kube root ca crt ConfigMapOptional lt nil gt DownwardAPI true QoS Class BestEffort Node Selectors lt none gt Tolerations op=Exists for 300s op=Exists for 300s Events Type Reason Age From Message Normal Scheduled 18m default scheduler Successfully assigned default py39 cacher to 856e6b497fcd Normal Pulling 18m kubelet Pulling image Normal Pulled 13m kubelet Successfully pulled image in 4m46 943337882s Normal Created 13m kubelet Created container flytekit Normal Started 13m kubelet Started container flytekit

Harry Souris · Answer

on image is still on pulling phase

Yee · Answer

`kubectl n flyte describe pod flyte sandbox xyzxyz xyzxyz`

Harry Souris · Answer

kubectl n flyte describe pod flyte sandbox b789778f6 fwkvq Name flyte sandbox b789778f6 fwkvq Namespace flyte Priority 0 Service Account flyte sandbox Node 856e6b497fcd 172 17 0 2 Start Time Sat 20 May 2023 00 41 50 +0300 Labels pod template hash=b789778f6 Annotations checksum cluster resource templates 6fd9b172465e3089fcc59f738b92b8dc4d8939360c19de8ee65f68b0e7422035 checksum configuration 7ef5ef618ebb04f965552e2e4814dc053ef5338fee3ada32517e4e4b1695989b checksum db password secret 669e1cdf4633c6dd40085f78d1bb6b9672d8120ff1f62077a879a4d46db133e2 Status Running IP 10 42 0 4 IPs IP 10 42 0 4 Controlled By ReplicaSet flyte sandbox b789778f6 Init Containers wait for db Container ID 8 Image bitnami postgresql sandbox Image ID sha256 a729f5f0de5fa39ba4d649e7366d499299304145d2456d60a16b0e63395bd61a Port lt none gt Host Port lt none gt Command sh ec Args until pg isready h flyte sandbox postgresql p 5432 U postgres do echo waiting for database sleep 0 1 done State Terminated Reason Completed Exit Code 0 Started Sat 20 May 2023 00 41 54 +0300 Finished Sat 20 May 2023 00 42 24 +0300 Ready True Restart Count 0 Environment lt none gt Mounts var run secrets kubernetes io serviceaccount from kube api access 9js4w ro Containers flyte Container ID 3 Image flyte binary sandbox Image ID sha256 b26c073652ff86b27f03a534177274b17dcb45f5ced24987c11282e5ddd7f110 Ports 8088 TCP 8089 TCP 9443 TCP Host Ports 0 TCP 0 TCP 0 TCP Args start config etc flyte config d yaml State Waiting Reason CrashLoopBackOff Last State Terminated Reason Error Exit Code 1 Started Sat 20 May 2023 01 00 02 +0300 Finished Sat 20 May 2023 01 00 02 +0300 Ready False Restart Count 9 Liveness http get http http healthcheck delay=0s timeout=1s period=10s success=1 failure=3 Readiness http get http http healthcheck delay=0s timeout=1s period=10s success=1 failure=3 Environment POD NAME flyte sandbox b789778f6 fwkvq v1 metadata name POD NAMESPACE flyte v1 metadata namespace Mounts etc flyte cluster resource templates from cluster resource templates rw etc flyte config d from config rw var run flyte from state rw var run secrets flyte db pass from db pass rw path= db pass var run secrets kubernetes io serviceaccount from kube api access 9js4w ro Conditions Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes cluster resource templates Type Projected a volume that contains injected data from multiple sources ConfigMapName flyte sandbox cluster resource templates ConfigMapOptional lt nil gt ConfigMapName flyte sandbox extra cluster resource templates ConfigMapOptional lt nil gt config Type Projected a volume that contains injected data from multiple sources ConfigMapName flyte sandbox config ConfigMapOptional lt nil gt ConfigMapName flyte sandbox extra config ConfigMapOptional lt nil gt db pass Type Secret a volume populated by a Secret SecretName flyte sandbox db pass Optional false state Type EmptyDir a temporary directory that shares a pod s lifetime Medium SizeLimit lt unset gt kube api access 9js4w Type Projected a volume that contains injected data from multiple sources TokenExpirationSeconds 3607 ConfigMapName kube root ca crt ConfigMapOptional lt nil gt DownwardAPI true QoS Class BestEffort Node Selectors lt none gt Tolerations op=Exists for 300s op=Exists for 300s Events Type Reason Age From Message Normal Scheduled 22m default scheduler Successfully assigned flyte flyte sandbox b789778f6 fwkvq to 856e6b497fcd Normal Pulled 22m kubelet Container image bitnami postgresql sandbox already present on machine Normal Created 22m kubelet Created container wait for db Normal Started 22m kubelet Started container wait for db Warning Unhealthy 21m x3 over 21m kubelet Liveness probe failed Get dial tcp 10 42 0 4 8088 connect connection refused Normal Killing 21m kubelet Container flyte failed liveness probe will be restarted Normal Pulled 20m x2 over 21m kubelet Container image flyte binary sandbox already present on machine Normal Created 20m x2 over 21m kubelet Created container flyte Normal Started 20m x2 over 21m kubelet Started container flyte Warning Unhealthy 17m x46 over 21m kubelet Readiness probe failed Get dial tcp 10 42 0 4 8088 connect connection refused Warning BackOff 2m23s x59 over 16m kubelet Back off restarting failed container

Yee · Answer

can you get all the logs for flyte binary container as well

Yee · Answer

` json src composite workqueue go 88 level debug msg Subqueue handler batch round ts 2023 05 19T21 53 48Z ` isn t quite enough need the messages from startup

Harry Souris · Answer

need to go now unfortunatelly

Harry Souris · Answer

< Yee> thanks for your help last night today I run 1 docker system prune a 2 started docker again and it worked not sure exactly with the prob last night perhaps a resource issue

Harry Souris · Answer

it worked in the sense i could access the UI and run a workflow at the demo cluster the execution of the wine dataset workflow fails though

Harry Souris · Answer

i think problem was with db 2023 05 20 19 22 36 738 GMT 266 LOG checkpoint starting end of recovery immediate wait 2023 05 20 19 22 36 752 GMT 266 LOG checkpoint complete wrote 3 buffers 0 0 0 WAL file s added 0 removed 0 recycled write=0 003 s sync=0 002 s total=0 014 s sync files=2 longest=0 001 s average=0 001 s distance=0 kB estimate=0 kB 2023 05 20 19 22 36 774 GMT 1 LOG database system is ready to accept connections 2023 05 20 19 27 10 073 GMT 1 LOG received smart shutdown request 2023 05 20 19 27 12 860 GMT 831 FATAL the database system is shutting down 2023 05 20 19 27 16 648 GMT 841 FATAL the database system is shutting down 2023 05 20 19 27 26 640 GMT 851 FATAL the database system is shutting down 2023 05 20 19 27 36 586 GMT 860 FATAL the database system is shutting down

Harry Souris · Answer

FYI < Yee>

Yee · Answer

why is the database shutting down can you `describe` the postgres pod

Yee · Answer

something feels wrong outside of flyte what type of system are you on how much resources have you given docker

Yee · Answer

may need to bump it up a bit

David Espejo (he/him) · Answer

< Harry Souris> what s the current status