https://flyte.org logo
Powered by
Title
a

Armaan Goel

09/20/2022, 10:44 PM
Still having issues with the flyte deployment actually, it seems like I can run jobs but they never complete!
Not sure how to debug this, because there are no logs
Looks like its stuck in queued
k

Ketan (kumare3)

09/20/2022, 11:04 PM
hmm
stuck in queued?
can you check the kubernetes namespace 
project-domain
kubectl get pods -n <namespace>
and see what the pods are, also in the UI
you should see a small 
i
icon that can show more details
ohh you have not installed flytepropeller
this is reallllllllly odd
you installation seems bad - phew!!!!!!!!!
cc @Shivay Lamba?
cc @Alireza can you help here?
a

Alireza

09/20/2022, 11:23 PM
Can you run this command and check the outputs: 
kubectl get pods -n flyte
The error points out that 
the server could not find the requested resources
I guess we need to review your installation process. where did you deploy 
FlyteAdmin
? i.e., GKE, EKS, on-prem cluster?
a

Armaan Goel

09/20/2022, 11:37 PM
GKE, yep
a

Alireza

09/20/2022, 11:41 PM
Did you follow the instruction on docs.flyte.org?
a

Armaan Goel

09/20/2022, 11:43 PM
Yes, but I had to make some modifications to get to this point
https://docs.google.com/document/d/1skJWmt3hJoIuPQr_RfR-gB9wlatVSIcSD5VlBylJqd8/edit documented here
a

Alireza

09/20/2022, 11:44 PM
can you also share the output of this command? 
kubectl get nodes -n flyte
is it ok if I open your doc?
Another Question. After reinstalling helm did you: 1. helm upgrade, or 2. helm install?
a

Armaan Goel

09/20/2022, 11:49 PM
Oh yeah feel free to check out the doc!
i did 
helm uninstall flyte -n flyte
then 
helm install -n flyte -f values-gcp.yaml --create-namespace flyte flyteorg/flyte-core
a

Alireza

09/20/2022, 11:51 PM
hmm, too less resources! I am just suggesting the cluster is so small. Not sure if it is related to error though!
helm install
is once! after that as pointed out in the doc to make modification only use 
helm upgrade
can you helm upgrade right now and see if the same error happens or not?
Meanwhile I take a look into your doc
Your installation process is correct! Good Job!
can you describe the failed pod?
a

Armaan Goel

09/21/2022, 1:28 AM
hmm which pod?
I figured it would be 
amt5m758z8xtv7rzqrnl-n0-0
but that doesn’t work
a

Alireza

09/21/2022, 1:29 AM
What is the output of this? 
kubectl get pods -n flyte
I am suspicous that your venv is breaking this. Is it possible to try it with conda?
a

Armaan Goel

09/21/2022, 1:35 AM
the venv of the code I’m deploying?
a

Alireza

09/21/2022, 1:35 AM
yeap
a

Armaan Goel

09/21/2022, 1:36 AM
I’m actually just deploying this flyte example
https://github.com/flyteorg/flytekit-python-template
a

Alireza

09/21/2022, 1:37 AM
(I guess) you are using a virtual environment, right?
Message: exec /opt/venv/bin/pyflyte-execute: exec format error
This error reflects that your encapsulated environment is failing. I don't think it has any to do with Flyte itself.
It would be helpful to send the process you made the virtual environment so that I can replicate the process. Or try it with conda virtual environment. something like this: 
conda create --name flyte python=3.9
conda activate flyte
a

Armaan Goel

09/21/2022, 1:47 AM
If you look at the error message I sent there do seem to be some grpc errors in flytepropellor
but also I dont think i’m using a venv I made a docker container based on the guide you had
i have the link to the github i reference just above and here is my version of it https://github.com/armaangoel78/flytetest
a

Alireza

09/21/2022, 1:55 AM
I see. Just a quick qustion do you know by any chance if your flyte config is setup correctly?
or how did you set it up?
a

Armaan Goel

09/21/2022, 1:59 AM
./flyte/config.yaml or the values file for the flyte cluster
a

Alireza

09/21/2022, 2:00 AM
./flyte/config.yaml
how did you set it up?
it seems there is a connection error
Please share your findings. I will discuss it with my colleagues. Sorry couldn't solve it all today. We'll check it tomorrow.
a

Armaan Goel

09/21/2022, 2:10 AM
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///flyte.gauntletai.com
  authType: Pkce
  insecure: false
logger:
  show-source: true
  level: 0%
a

Alireza

09/21/2022, 2:20 AM
I got it
a

Armaan Goel

09/21/2022, 4:08 AM
@Alireza no worries thanks so much for you help it means a ton 🙂
4 Views
#flyte-deploymentJoin Slack
Powered by