https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
s

Samuel Bentley

04/25/2023, 2:36 PM
Hey everyone, I'm trying to run a wf on my AWS single binary deployment of Flyte. I've got port forwarding running with
Copy code
kubectl -n flyte port-forward service/flyte-backend-flyte-binary 8088:8088 8089:8089
Config is this
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: localhost:8088
  authType: Pkce
  insecure: true
logger:
  show-source: true
  level: 0
And I'm getting this when running a wf:
Copy code
$ FLYTE_CONFIG=~/.flyte/config.yaml pyflyte run --remote workflows/greeting.py wf --name Sam
Failed with Exception: Reason: SYSTEM:Unknown
RPC Failed, with Status: StatusCode.UNAVAILABLE
	details: failed to connect to all addresses; last error: INTERNAL: ipv4:127.0.0.1:8088: Trying to connect an http1.x server
	Debug string UNKNOWN:failed to connect to all addresses; last error: INTERNAL: ipv4:127.0.0.1:8088: Trying to connect an http1.x server {grpc_status:14, created_time:"2023-04-25T15:31:54.782127+01:00"}
d

David Espejo (he/him)

04/25/2023, 2:43 PM
Hi @Samuel Bentley Try adding 
insecureSkipVerify: true
s

Samuel Bentley

04/25/2023, 3:39 PM
Like this?
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: localhost:8088
  authType: Pkce
  insecure: true
  insecureSkipVerify: true
logger:
  show-source: true
  level: 0
Still no luck :-(
d

David Espejo (he/him)

04/25/2023, 4:20 PM
and change 
insecure
to 
false
s

Samuel Bentley

04/25/2023, 4:22 PM
Now I get this :-(
Copy code
Failed with Unknown Exception <class 'ssl.SSLError'> Reason: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1129)
[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1129)
d

David Espejo (he/him)

04/25/2023, 4:38 PM
is this running on the cloud?
y

Yee

04/25/2023, 6:30 PM
8088 is http, for flytekit, use 8089
insecureskipverify doesn’t do anything if insecure is true
and insecure should be true.
d

David Espejo (he/him)

04/25/2023, 6:46 PM
right
I'll have a session with Samuel but meanwhile @Samuel Bentley 1. Remove the 
insecureSkipVerify
line 2. Leave 
insecure: true
3. Change the port from 
8088
to 
8089
s

Samuel Bentley

04/25/2023, 7:20 PM
Cool, that's "working" now. @David Espejo (he/him) I'm back to the 403 error now. Catch up tomorrow :-)
Thanks @Yee 🙂
s

Sidharth(Sid)

05/24/2023, 9:40 AM
Hi guys. Similar issue for me as well. With some WARNINGS first,
Copy code
~/data_ops/eks/flyte/flytesnacks/cookbook$ pyflyte run --remote core/flyte_basics/hello_world.py my_wf 

2023-05-24 15:01:01,173936 WARNING  {"asctime": "2023-05-24 15:01:01,173", "name": "flytekit", "levelname": "WARNING", "message": "Error while parsing a block mapping\n  in     file.py:185
                                    \"/home/enceladus/.flyte/config.yaml\", line 1, column 1\nexpected <block end>, but found '<block mapping start>'\n  in
                                    \"/home/enceladus/.flyte/config.yaml\", line 4, column 3 reading yaml config file at /home/enceladus/.flyte/config.yaml, ignoring..."}
2023-05-24 15:01:01,217704 WARNING  {"asctime": "2023-05-24 15:01:01,217", "name": "flytekit", "levelname": "WARNING", "message": "Error while parsing a block mapping\n  in     file.py:185
                                    \"/home/enceladus/.flyte/config.yaml\", line 1, column 1\nexpected <block end>, but found '<block mapping start>'\n  in
                                    \"/home/enceladus/.flyte/config.yaml\", line 4, column 3 reading yaml config file at /home/enceladus/.flyte/config.yaml, ignoring..."}
2023-05-24 15:01:01,221089 WARNING  {"asctime": "2023-05-24 15:01:01,221", "name": "flytekit", "levelname": "WARNING", "message": "Error while parsing a block mapping\n  in     file.py:185
                                    \"/home/enceladus/.flyte/config.yaml\", line 1, column 1\nexpected <block end>, but found '<block mapping start>'\n  in
                                    \"/home/enceladus/.flyte/config.yaml\", line 4, column 3 reading yaml config file at /home/enceladus/.flyte/config.yaml, ignoring..."}
2023-05-24 15:01:02,057700 WARNING  {"asctime": "2023-05-24 15:01:02,057", "name": "flytekit", "levelname": "WARNING", "message": "Error while parsing a block mapping\n  in     file.py:185
                                    \"/home/enceladus/.flyte/config.yaml\", line 1, column 1\nexpected <block end>, but found '<block mapping start>'\n  in
                                    \"/home/enceladus/.flyte/config.yaml\", line 4, column 3 reading yaml config file at /home/enceladus/.flyte/config.yaml, ignoring..."}
2023-05-24 15:01:02,060970 WARNING  {"asctime": "2023-05-24 15:01:02,060", "name": "flytekit", "levelname": "WARNING", "message": "Error while parsing a block mapping\n  in     file.py:185
                                    \"/home/enceladus/.flyte/config.yaml\", line 1, column 1\nexpected <block end>, but found '<block mapping start>'\n  in
                                    \"/home/enceladus/.flyte/config.yaml\", line 4, column 3 reading yaml config file at /home/enceladus/.flyte/config.yaml, ignoring..."}


Failed with Exception Code: SYSTEM:Unknown
RPC Failed, with Status: StatusCode.UNAVAILABLE
        details: failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:30080: Failed to connect to remote host: Connection refused
        Debug string UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:30080: Failed to connect to remote host: Connection refused {created_time:"2023-05-24T15:01:09.752070056+05:30", grpc_status:14}
heres the config -
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
        endpoint: dns:///localhost:8089
  authType: Pkce
  insecure: true
logger:
  show-source: true
  level: 0
I installed single cloud cluster flyte with latest 1.6.1 helm chart. both http and grpc are forwarded in other terminals. The flyte frontend is up on 8088. any thoughts?
d

David Espejo (he/him)

05/24/2023, 3:06 PM
@Sidharth(Sid) have you ever used 
flytectl demo
in your system? From the error message, it's weird to see pyflyte trying to connect to 
30080
, the port for sandbox
s

Sidharth(Sid)

05/25/2023, 5:55 AM
hmm, i think I have a long time ago. is there something I should clear/remove?
d

David Espejo (he/him)

05/30/2023, 6:20 PM
sorry @Sidharth(Sid) yeah you could 
flytectl demo teardown
and it will remove your local sandbox instance. Then port-forward your flyte-binary and try again please
y

Yee

05/30/2023, 8:06 PM
flytectl demo teardown --volume
to remove the local db volume also
s

Sidharth(Sid)

05/31/2023, 6:54 AM
now the console isn't coming up at localhost8088. There are no errors or any other logs in the http port forward terminal, its just showing
Copy code
Forwarding from 127.0.0.1:8088 -> 8088
Forwarding from [::1]:8088 -> 8088
, and in config, ive kept
Copy code
admin:
        endpoint: dns:///localhost:8088
  authType: Pkce
  insecure: true
logger:
  show-source: true
  level: 0
y

Yee

05/31/2023, 4:31 PM
localhost:30080
d

David Espejo (he/him)

05/31/2023, 4:45 PM
I think @Sidharth(Sid) is running 
flyte-binary
is that correct?
y

Yee

05/31/2023, 4:46 PM
oh you’re port forwarding a full helm install
oh your endpoint should be 8089
not 8088
8088 is http, 8089 is grpc
s

Sidharth(Sid)

06/01/2023, 6:26 AM
yes im at the stage of flyte-binary in AWS
ok will try. also will talk my colleagues here, there are many new security related changes going on, might be a AWS security group thats blocking me.
3 Views
Powered by