https://flyte.org
Join Slack
Still having issues with the flyte deployment actually, it seems like I can run jobs but they never ...
# flyte-deployment
s
Still having issues with the flyte deployment actually, it seems like I can run jobs but they never complete!
Not sure how to debug this, because there are no logs
Looks like its stuck in queued
f
hmm
stuck in queued?
can you check the kubernetes namespace 
project-domain
kubectl get pods -n <namespace>
and see what the pods are, also in the UI
you should see a small 
i
icon that can show more details
s
while debugging, I thought I might 
helm uninstall
flyte and reinstall it, but now there’s a brand new error
f
ohh you have not installed flytepropeller
this is reallllllllly odd
you installation seems bad - phew!!!!!!!!!
cc @victorious-park-53030?
cc @dry-teacher-15656 can you help here?
➕ 1
d
Can you run this command and check the outputs: 
kubectl get pods -n flyte
s
everything seems to be running
👍 1
d
The error points out that 
the server could not find the requested resources
I guess we need to review your installation process. where did you deploy 
FlyteAdmin
? i.e., GKE, EKS, on-prem cluster?
s
GKE, yep
👍 1
d
Did you follow the instruction on docs.flyte.org?
s
Yes, but I had to make some modifications to get to this point
👍 1
d
can you also share the output of this command? 
kubectl get nodes -n flyte
is it ok if I open your doc?
Another Question. After reinstalling helm did you: 1. helm upgrade, or 2. helm install?
s
message has been deleted
Oh yeah feel free to check out the doc!
i did 
helm uninstall flyte -n flyte
then 
helm install -n flyte -f values-gcp.yaml --create-namespace flyte flyteorg/flyte-core
d
hmm, too less resources! I am just suggesting the cluster is so small. Not sure if it is related to error though!
helm install
is once! after that as pointed out in the doc to make modification only use 
helm upgrade
can you helm upgrade right now and see if the same error happens or not?
Meanwhile I take a look into your doc
Your installation process is correct! Good Job!
s
great that worked, new error now
d
can you describe the failed pod?
s
hmm which pod?
I figured it would be 
amt5m758z8xtv7rzqrnl-n0-0
but that doesn’t work
d
What is the output of this? 
kubectl get pods -n flyte
s
message has been deleted
hmm getting some interesting logging in flyte propeller
d
I am suspicous that your venv is breaking this. Is it possible to try it with conda?
s
the venv of the code I’m deploying?
d
yeap
s
I’m actually just deploying this flyte example
d
(I guess) you are using a virtual environment, right?
Copy code
Message: exec /opt/venv/bin/pyflyte-execute: exec format error
This error reflects that your encapsulated environment is failing. I don't think it has any to do with Flyte itself.
It would be helpful to send the process you made the virtual environment so that I can replicate the process. Or try it with conda virtual environment. something like this: 
conda create --name flyte python=3.9
conda activate flyte
s
If you look at the error message I sent there do seem to be some grpc errors in flytepropellor
👍 1
but also I dont think i’m using a venv I made a docker container based on the guide you had
i have the link to the github i reference just above and here is my version of it https://github.com/armaangoel78/flytetest
👍 1
d
I see. Just a quick qustion do you know by any chance if your flyte config is setup correctly?
or how did you set it up?
s
./flyte/config.yaml or the values file for the flyte cluster
d
./flyte/config.yaml
how did you set it up?
it seems there is a connection error
Please share your findings. I will discuss it with my colleagues. Sorry couldn't solve it all today. We'll check it tomorrow.
s
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///flyte.gauntletai.com
  authType: Pkce
  insecure: false
logger:
  show-source: true
  level: 0%
d
I got it
s
@dry-teacher-15656 no worries thanks so much for you help it means a ton 🙂
👍 1
183 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by