https://flyte.org
Join Slack
h
message has been deleted
t
cc: @icy-agent-73298 @great-school-54368, @hundreds-intern-44546 isn’t able to start a demo cluster on Ubuntu and RHEL. 1. On Ubuntu, the above screenshot explains the error. 2. On RHEL, the error is 
context default doesn't exist
. Reference:

https://flyte-org.slack.com/files/U044UT2LV8C/F044KRZG4NS/image.png▾

i
1. Can we get docker logs from the failed container. 2. Seems it failed in the last step when copying the sandbox context to local machines context and switch it over. So can we check is locally we have kube config . Atleast a local kube client is expected to be there before using sandbox or sandbox-lite
h
yes sure will post you soon
hello @icy-agent-73298 here are the logs for the same
message has been deleted
will install local kube client and let you know current status thanks for the reply
i
Not much we can get from those logs. We have ticket to collect diagnostic info to help more on this https://github.com/flyteorg/flyte/issues/2192 . But on RHEL your logs show it went forward but failed in last step so do let us know once you have local kube client .
👍 1
h
hello @icy-agent-73298 its working when i have installed the kubernetes in backend...
Thanks Team for support!
hello @icy-agent-73298 i am trying to start sandbox environment on k8s but minio goes in crashloobbackoff again and again could you please help me to find what's getting wrong .here...
result from describe command for minio
Copy code
Tolerations:                 <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                             <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  16m                 default-scheduler  Successfully assigned flyte/minio-5745464bb-zthw8 to ip-172-31-31-189.ap-south-1.compute.internal
  Normal   Pulling    16m                 kubelet            Pulling image "<http://ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0|ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0>"
  Normal   Pulled     16m                 kubelet            Successfully pulled image "<http://ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0|ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0>" in 13.816089274s
  Normal   Created    14m (x5 over 16m)   kubelet            Created container minio
  Normal   Started    14m (x5 over 16m)   kubelet            Started container minio
  Normal   Pulled     14m (x4 over 16m)   kubelet            Container image "<http://ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0|ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0>" already present on machine
  Warning  BackOff    85s (x61 over 15m)  kubelet            Back-off restarting failed container
and result from logs
Copy code
[root@ip-172-31-31-189 ~]# kubectl logs  minio-5745464bb-zthw8 -n flyte
 07:15:01.89
 07:15:01.90 Welcome to the Bitnami minio container
 07:15:01.90 Subscribe to project updates by watching <https://github.com/bitnami/bitnami-docker-minio>
 07:15:01.90 Submit issues and feature requests at <https://github.com/bitnami/bitnami-docker-minio/issues>
 07:15:01.90
 07:15:01.90 INFO  ==> ** Starting MinIO setup **
 07:15:02.01 INFO  ==> Starting MinIO in background...
 07:15:12.01 INFO  ==> Adding local Minio host to 'mc' configuration...
 07:15:12.41 INFO  ==> MinIO is already stopped...
message has been deleted
i
Strange we haven’t seen this issue . Can you check this one from bitnami issues https://github.com/bitnami/bitnami-docker-minio/issues/3 . Which platform are you running this on and did you ever get minio pod running
h
nope this is not working
cause i have followed helm chart to install and not any dokcer container.. so directly run the commands which was given in https://docs.flyte.org/en/latest/deployment/sandbox.html#deployment-sandbox
i
You can change to the latest minio image directly and restart the minio pod and see if that fixes the issue. Flyte just reuses the minio community project . Might be the version flyte comes with is not compatible on your platform and hence you can check with latest image and see if that helps. Also if it still fails, would be worth commenting on the issue i posted so that minio community can help more on it too
h
so which means i need to change the helm chart conf right
i
easy way would be to just edit the deployment . if that works then you can edit the helm chart conf to persist this info https://github.com/flyteorg/flyte/blob/master/charts/flyte/values.yaml#L723
h
what will be the value for the same..
for edit
hello I have logs for postgres as well might be this will helpful....
Copy code
PostgreSQL init process complete; ready for start up.

2022-10-08 09:19:24.525 UTC [1] LOG:  starting PostgreSQL 13.5 (Ubuntu 13.5-0ubuntu0.21.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0, 64-bit
2022-10-08 09:19:24.526 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2022-10-08 09:19:24.526 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2022-10-08 09:19:24.528 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-10-08 09:19:24.533 UTC [66] LOG:  database system was shut down at 2022-10-08 09:19:24 UTC
2022-10-08 09:19:24.539 UTC [1] LOG:  database system is ready to accept connections
2022-10-08 09:20:37.000 UTC [74] ERROR:  relation "projects" does not exist at character 15
2022-10-08 09:20:37.000 UTC [74] STATEMENT:  SELECT * FROM "projects" WHERE state <> $1 ORDER BY created_at desc
2022-10-08 09:20:51.434 UTC [75] FATAL:  database "datacatalog" does not exist
pleaes convey this to team so that will run smooth.. I was trying installation from long time but it's getting failed multiple times so.. if possible please let someone help me in the process
i
Are you available for a call today. We can check what’s happening over google meet.Let me know if that works or Tuesday .
h
yes I will be available what will be your preferred time??
i
How about 8 pm IST
h
sure i will be available
👍 1
hello please join
i
159 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by