https://flyte.org logo
Join Slack
Powered by
New RunLLM response generated: >>>*# Question* Hi...
# runllm-flyte-trial
a
New RunLLM response generated: >>>*# Question* Hi Everyone ... I am receiving following error in flyte.
Copy code
Running Execution on Remote.
E0805 10:34:02.379989450 3110954 <http://ssl_transport_security.cc:1653]|ssl_transport_security.cc:1653]>       Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.
RPC Failed, with Status: StatusCode.UNAVAILABLE
        details: failed to connect to all addresses; last error: UNKNOWN: ipv4:10.6.74.230:443: Ssl handshake failed: SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
        Debug string UNKNOWN:Error received from peer  {grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:10.6.74.230:443: Ssl handshake failed: SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED", grpc_status:14, created_time:"2024-08-05T10:34:02.380439646+02:00"}
<_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:10.6.74.230:443: Ssl handshake failed: SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:10.6.74.230:443: Ssl handshake failed: SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED", grpc_status:14, created_time:"2024-08-05T10:34:02.380439646+02:00"}"
My Flyteconsole is working fine but pyflyte isn't working as it was before. I changed the environment and created new one and install flytekit in it. and after that I am receiving this error. In previous environment , Everything is working fine, I mean authentication is done successfully at every end and no error logs i see in any pods but it still didn't generate the flyteconsole link after running pyflyte run --remote workflows/main.py wf command. It just quit after Running Execution on Remove. Can you please tell me if anything I can do in solving it and where should I start my debugging process # Answer The error you're encountering is related to SSL certificate verification failure when using 
pyflyte
. This issue is common when using self-signed certificates or certificates not trusted by the Python 
grpc
client. Steps to Resolve: 1. Verify Certificate Configuration: Ensure that your Flyte configuration includes the correct paths to your CA certificate. You can set the 
caCertFilePath
in your Flyte configuration file (
~/.flyte/config.yaml
).
Copy code
admin:
     endpoint: "dns:///your.flyte.endpoint"
     insecure: false
     insecureSkipVerify: true
     caCertFilePath: /path/to/your/ca-bundle.crt
>>
2. Set Environment Variable: Set the 
REQUESTS_CA_BUNDLE
environment variable to the path of your CA certificate file. This will make the Python 
requests
library trust your self-signed certificate.
Copy code
export REQUESTS_CA_BUNDLE=/path/to/your/ca-bundle.crt
3. Use Non-SSL Endpoint: If possible, configure Flyte to use a non-SSL endpoint by setting 
admin.insecure
to 
true
in your Flyte configuration file.
Copy code
admin:
     endpoint: "dns:///your.flyte.endpoint"
     insecure: true
4. Check Subject Alternative Name (SAN): Ensure that the IP address or DNS name you are connecting to is included in the SAN of your certificate. This is crucial for SSL verification.
Copy code
[alt_names]
   DNS.1 = your.flyte.endpoint
   IP.1 = 10.6.74.230
Example Configuration:
Copy code
admin:
  endpoint: "dns:///your.flyte.endpoint"
  insecure: false
  insecureSkipVerify: true
  caCertFilePath: /path/to/your/ca-bundle.crt
logger:
  show-source: true
  level: 6
Debugging Steps: 1. Check Logs: Review the logs of the Flyte Admin and Flyte Propeller pods for any SSL-related errors. 2. Test with 
flytectl
: Use 
flytectl
to verify if the issue is specific to 
pyflyte
or a general SSL issue.
Copy code
flytectl get projects
3. Verify Certificate: Use tools like 
openssl
to verify the certificate chain and ensure it is correctly configured.
Copy code
openssl s_client -connect your.flyte.endpoint:443 -CAfile /path/to/your/ca-bundle.crt
Would you like more detailed steps or further assistance with any specific part of the process? Sources: - Flyte Issue #4794 - Flyte Discussion on SSL Issues Slack Link https://flyte-org.slack.com/archives/CP2HDHKE1/p1722847475.459999 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
2 Views
Open in Slack
PreviousNext
Powered by