https://flyte.org
Join Slack
hi, i've used the eks-starter.yaml values file from the docs to deploy flyte to my eks cluster, but ...
# flyte-deployment
b
hi, i've used the eks-starter.yaml values file from the docs to deploy flyte to my eks cluster, but when i try to do the verify installation section of the docs, when i forward port 8089 and then try running pyflyte --run hello_world,py hello_world_wf i see the following output in my console:
Copy code
> kubectl -n flyte port-forward service/flyte-backend-flyte-binary-grpc 8089:8089     Wed May 29 10:38:33 2024
Forwarding from 127.0.0.1:8089 -> 8089
Forwarding from [::1]:8089 -> 8089
Handling connection for 8089
Handling connection for 8089
E0529 10:38:40.778681   46155 portforward.go:398] error copying from local connection to remote stream: writeto tcp6 [::1]:8089->[::1]:53207: read tcp6 [::1]:8089->[::1]:53207: read: connection reset by peer
Handling connection for 8089
and no executions are registered on the flyte console i have running in my cluster. any advice on how to debug this?
a
Hi @bumpy-needle-36220 The error message seems to point to a loopback IPv6 address. Is IPv6 enabled in your network card?
does the UI connection work?
b
yes the ui connection works when i port forward 8088
in one terminal i have 
kubectl -n flyte port-forward service/flyte-backend-flyte-binary-http 8088:8088
in the other 
kubectl -n flyte port-forward service/flyte-backend-flyte-binary-grpc 8089:8089
the docs say that i should run 
kubectl -n flyte port-forward service/flyte-binary 8088:8088 8089:8089
but when i try that i get 
Error from server (NotFound): services "flyte-binary" not found
otherwise ive deployed flyte using the helm chart exactly as the single cluster instructions in the docs specify
and yes, im using an rds database with dual stack networking enabled so ipv6 should work
a
you're right, that command doesn't work with the default config
b
both the eks cluster and rds are in the same vpc so networking between them should be fine
a
What are you using as 
endpoint
in your local 
config.yaml
?
b
Copy code
> cat ~/.flyte/config.yaml                                                            Wed May 29 11:09:27 2024
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///localhost:8089
  insecure: true
  authType: Pkce
logger:
  show-source: true
  level: 0
a
try removing the 
dns:///
prefix
b
no dice. i removed the prefix, restarted both port-forwards, then tried 
venv > pyflyte run --remote hello_world.py hello_world_wf
and still got
Copy code
> kubectl -n flyte port-forward service/flyte-backend-flyte-binary-grpc 8089:8089     Wed May 29 11:11:34 2024
Forwarding from 127.0.0.1:8089 -> 8089
Forwarding from [::1]:8089 -> 8089
Handling connection for 8089
E0529 11:11:42.471694   53429 portforward.go:398] error copying from local connection to remote stream: writeto tcp6 [::1]:8089->[::1]:53850: read tcp6 [::1]:8089->[::1]:53850: read: connection reset by peer
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: localhost:8089
  insecure: true
  authType: Pkce
logger:
  show-source: true
  level: 0
wonder why the remote server is resetting the tcp connection. wonder if its a k8s problem. when i run 
kubectl -n flyte logs <podname>
i dont see anything in the logs that would suggest flyte even got the request
a
is there any firewall between your machine and the cluster?
b
shouldn't be
if it were the case wouldn't the ui connection be dropped too
👍🏽 1
ui works fine
a
can you increase 
logger.level
to 
6
?
b
no change from that either unfortunately. where would you expect to see the additional log output? pyflyte just says "running execution on remote" and nothing more
a
it would be on the client but apparently Flyte is effectively not receiving a request
b
yeah...
my eks cluster is ipv6-only otherwise i'd try forcing it to connect over ipv4
actually that fails too:
Copy code
E0529 11:20:37.069807   54098 portforward.go:398] error copying from local connection to remote stream: writeto tcp4 127.0.0.1:8089->127.0.0.1:53927: read tcp4 127.0.0.1:8089->127.0.0.1:53927: read: connection reset by peer
E0529 11:20:37.069966   54098 portforward.go:398] error copying from local connection to remote stream: writeto tcp6 [::1]:8089->[::1]:53926: read tcp6 [::1]:8089->[::1]:53926: read: connection reset by peer
any thoughts?
a
not really sure what's happening here
b
here's something else i tried. i used kubectl run to spawn an ubuntu:24.04 container inside the flyte namespace. (called debug2)
Copy code
> kubectl -n flyte get pods                                                           Wed May 29 13:17:02 2024
NAME                                          READY   STATUS    RESTARTS   AGE
debug2                                        1/1     Running   0          11m
flyte-backend-flyte-binary-578758d8c9-zpnwt   1/1     Running   0          161m
inside this container, i installed python/pip, flytectl, and flytekit. then, i set my 
.flyte/config.yaml
to this:
Copy code
(venv) root@debug2:~# cat .flyte/config.yaml
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: flyte-backend-flyte-binary-grpc:8089
  insecure: true
logger:
  level: 6
  show-source: true
still, when i run 
pyflyte run --remote hello_world.py hello_world_wf
i get 
Running Execution on Remote
but nothing shows up in the UI console
there should be no networking issues inside the namespace
a
uff, and no relevant logs on the 
flyte-backend.
.. Pod right?
b
yep, nothing i can find
it's just saying "exiting subqueue handler batch round"
i followed the instructions from https://github.com/davidmirror-ops/flyte-the-hard-way/blob/main/docs/aws/05-deploy-with-helm.md from scratch, created a new eks cluster, new rds, new everything, still can't get workflows to run remotely
a
hey @bumpy-needle-36220 so still same error?
b
yep
a
have you tried adding an Ingress? not sure if the port-forward is getting lost somewhere
62 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by