#3330 [BUG] pyflyte run results in error: Handshake failed with fatal error OPENSSL_internal:WRONG_VERSION_NUMBER while connecting to flyte-binary in k8s Issue created by abuvaneswari Describe the bug I followed the instructions and hosted flyte-binary in a Kubernetes cluster, v 1.20 (tried on 1.21 as well): https://docs.flyte.org/en/latest/deployment/deployment/cloud_simple.html I attach the values.yaml file that was used in the helm install command. values.yaml.txt I verified that I am able to access the flyte console (as I have configured NodePort. when I port-forward also, the same error occurs). Verified that the following curl command works: $ curl http://135.104.238.138:8088/api/v1/projects {"projects":[{"id":"flytesnacks","name":"flytesnacks","domains":[{"id":"development","name":"development"},{"id":"staging","name":"staging"},{"id":"production","name":"production"}],"description":"flytesnacks description"}]} Attempted pyflyte run command with this remote and got the following error: (Note that the same command runs fine when run in the 'demo' environment) $ pyflyte --config ~/.flyte/config.yaml run --remote example1.py training_workflow --hyperparameters '{"C": 0.1}' E0212 021723.386151523 3823 ssl_transport_security.cc:1501] Handshake failed with fatal error SSL_ERROR_SSL: error100000f7SSL routinesOPENSSL internalWRONG_VERSION_NUMBER. {"asctime": "2023-02-12 021723,412", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = "failed to connect to all addresses; last error: UNKNOWN: ipv4135.104.238.1388088: Ssl handshake failed: SSL_ERROR_SSL: error100000f7SSL routinesOPENSSL internalWRONG_VERSION_NUMBER"\n\tdebug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4135.104.238.1388088: Ssl handshake failed: SSL_ERROR_SSL: error100000f7SSL routinesOPENSSL internalWRONG_VERSION_NUMBER {grpc_status:14, created_time:"2023-02-12T021723.412065046-05:00"}"\n>, sleeping 200ms and retrying"} {"asctime": "2023-02-12 021723,613", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = "failed to connect to all addresses; last error: UNKNOWN: ipv4135.104.238.1388088: Ssl handshake failed: SSL_ERROR_SSL: error100000f7SSL routinesOPENSSL internalWRONG_VERSION_NUMBER"\n\tdebug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4135.104.238.1388088: Ssl handshake failed: SSL_ERROR_SSL: error100000f7SSL routinesOPENSSL internalWRONG_VERSION_NUMBER {grpc_status:14, created_time:"2023-02-12T021723.613242602-05:00"}"\n>, sleeping 400ms and retrying"} Traceback (most recent call last): File "/home/buvana/anaconda3/envs/flyte_qs/bin/pyflyte", line 8, in sys.exit(main()) File "/home/buvana/anaconda3/envs/flyte_qs/lib/python3.8/site-packages/click/core.py", line 1130, in call return self.main(*args, **kwargs) File "/home/buvana/anaconda3/envs/flyte_qs/lib/python3.8/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/home/buvana/anaconda3/envs/flyte_qs/lib/python3.8/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/buvana/anaconda3/envs/flyte_qs/lib/python3.8/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/buvana/anaconda3/envs/flyte_qs/lib/python3.8/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/buvana/anaconda3/envs/flyte_qs/lib/python3.8/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/buvana/anaconda3/envs/flyte_qs/lib/python3.8/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "/home/buvana/anaconda3/envs/flyte_qs/lib/python3.8/site-packages/flytekit/clis/sdk_in_container/run.py", line 552, in _run remote_entity = remote.register_script( File "/home/buvana/anaconda3/envs/flyte_qs/lib/python3.8/site-packages/flytekit/remote/remote.py", line 800, in register_script upload_location, md5_bytes = fast_register_single_script( File "/home/buvana/anaconda3/envs/flyte_qs/lib/python3.8/site-packages/flytekit/tools/script_mode.py", line 111, in fast_register_single_script upload_location = create_upload_location_fn(content_md5=md5) File "/home/buvana/anaconda3/envs/flyte_qs/lib/python3.8/site-packages/flytekit/clients/friendly.py", line 998, in get_upload_signed_url return super(SynchronousFlyteClient, self).create_upload_location( File "/home/buvana/anaconda3/envs/flyte_qs/lib/python3.8/site-packages/flytekit/clients/raw.py", line 43, in handler return fn(*args, **kwargs) File "/home/buvana/anaconda3/envs/flyte_qs/lib/python3.8/site-packages/flytekit/clients/raw.py", line 871, in create_upload_location return self._dataproxy_stub.CreateUploadLocation(create_upload_location_request, metadata=self._metadata) File "/home/buvana/anaconda3/envs/flyte_qs/lib/python3.8/site-packages/grpc/_channel.py", line 946, in call return _end_unary_response_blocking(state, call, False, None) File "/home/buvana/anaconda3/envs/flyte_qs/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "failed to connect to all addresses; last error: UNKNOWN: ipv4135.104.238.1388088: Ssl handshake failed: SSL_ERROR_SSL: error100000f7SSL routinesOPENSSL internalWRONG_VERSION_NUMBER" debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4135.104.238.1388088: Ssl handshake failed: SSL_ERROR_SSL: error100000f7SSL routinesOPENSSL internalWRONG_VERSION_NUMBER {grpc_status:14, created_time:"2023-02-12T021724.014732644-05:00"}" Expected behavior flyte workflow should be deployed in the remote and got executed Additional context to reproduce flyte-binary running in k8s cluster 1.20 (and 1.21) openssl version command when executed within the flyte-binary pod produced 1.1.1s as the version client: flytekit==1.3.2 python 3.8 (same error occurs in 3.7 and 3.9) OS: Linux and Windows (error occurs in both) openssl version: 1.1.1s and 1.1.1n ~/.flyte/config.yaml file's contents: admin: endpoint: dns///135.104.238.1388088 authType: Pkce insecure: false logger: show-source: true level: 0 Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte