HIMANSHU JOSHI
02/07/2023, 1:28 PMgrpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.25.0.46:443: Ssl handshake failed: SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED"
debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.25.0.46:443: Ssl handshake failed: SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED {grpc_status:14, created_time:"2023-02-07T18:47:06.951145+05:30"}"
Any body has an idea what is the problem here?
P.s I have TLS disabled on ingress.Bernhard Stadlbauer
02/07/2023, 1:39 PMadmin.insecure: True
set in your ~/.flyte/config.yaml
?HIMANSHU JOSHI
02/07/2023, 1:41 PMadmin:
endpoint: "dns:///"
# authType: Pkce
insecure: false
insecureSkipVerify: true
console:
endpoint: ""
logger:
show-source: true
level: 0
flytectl get projects
Bernhard Stadlbauer
02/07/2023, 1:55 PMHIMANSHU JOSHI
02/07/2023, 2:27 PMDavid Espejo (he/him)
02/07/2023, 3:10 PMHIMANSHU JOSHI
02/07/2023, 4:09 PMKetan (kumare3)
HIMANSHU JOSHI
02/07/2023, 4:30 PMName: grpcio
Version: 1.51.1
Summary: HTTP/2-based RPC framework
Home-page: <https://grpc.io>
Author: The gRPC Authors
Author-email: <mailto:grpc-io@googlegroups.com|grpc-io@googlegroups.com>
License: Apache License 2.0
Location: "<package-path>"
Requires:
Required-by: flytekit, grpcio-status
Eduardo Apolinario (eapolinario)
02/07/2023, 4:45 PMSujith Samuel
02/07/2023, 4:49 PMHIMANSHU JOSHI
02/07/2023, 5:04 PMAnkit Goyal
02/07/2023, 5:17 PMSujith Samuel
02/07/2023, 5:17 PMHIMANSHU JOSHI
02/07/2023, 5:18 PMAnkit Goyal
02/07/2023, 5:19 PMhttps
, can you click on the not secure
certificate is not valid
HIMANSHU JOSHI
02/07/2023, 5:19 PMAnkit Goyal
02/07/2023, 5:20 PMHIMANSHU JOSHI
02/07/2023, 5:20 PMAnkit Goyal
02/07/2023, 5:21 PMIssue by
-> Common Name (CN)
say?HIMANSHU JOSHI
02/07/2023, 5:21 PMAnkit Goyal
02/07/2023, 5:22 PMHIMANSHU JOSHI
02/07/2023, 5:23 PMAnkit Goyal
02/07/2023, 5:23 PMHIMANSHU JOSHI
02/07/2023, 5:24 PMAnkit Goyal
02/07/2023, 5:26 PMadmin:
caCertFilePath: /path/to/ca-bundle.crt
HIMANSHU JOSHI
02/07/2023, 5:27 PMAnkit Goyal
02/07/2023, 5:27 PMHIMANSHU JOSHI
02/07/2023, 5:30 PMAnkit Goyal
02/07/2023, 5:31 PMHIMANSHU JOSHI
02/07/2023, 5:33 PMAnkit Goyal
02/07/2023, 5:33 PMadmin.insecure: True
HIMANSHU JOSHI
02/08/2023, 12:18 PMpyflyte package
i tried it and it throws an error i.e. submodule not found?
P.s We're still not able to resolve ssl issue so can't use pyflyte register bcz of thatSamhita Alla
pyflyte package
should work. Could you share the exact error?HIMANSHU JOSHI
02/08/2023, 12:20 PMSamhita Alla
pyflyte package
contain utils
?HIMANSHU JOSHI
02/08/2023, 12:22 PMpyflyte --pkgs dynamic_workflow package --image <image-name> -f -o dynamic_workflow.tgz
command for referenceDoes the package you're sending toyescontainpyflyte package
?utils
Samhita Alla
from .utils ...
HIMANSHU JOSHI
02/08/2023, 12:50 PMError when trying to reconcile workflow
and in the dynamic task it was suppose to create 5 different task as i have given 5 input it did create those but it's been 20 min it is still stuck in queued and propeller is throwing above mentioned error and only one pod for dynamic task was created expecting 5Bernhard Stadlbauer
02/08/2023, 12:52 PMrpc error: code = InvalidArgument desc = missing project
somewhere in your propeller logs?HIMANSHU JOSHI
02/08/2023, 12:52 PM[Feb 8, 2023 6:20:04 PM GMT+5]
{"json":{"exec_id":"ahlrb9l95q2h4gg8fhgf","node":"n0/dn0","ns":"development","res_ver":"55914809","routine":"worker-14","wf":"flyteexamples:development:dynamic_workflow.main.dynamic_workflow"},"level":"error","msg":"handling parent node failed with error: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]","ts":"2023-02-08T12:50:04Z"}
[Feb 8, 2023 6:20:04 PM GMT+5]
{"json":{"exec_id":"ahlrb9l95q2h4gg8fhgf","node":"n0/dn0","ns":"development","res_ver":"55914809","routine":"worker-14","wf":"flyteexamples:development:dynamic_workflow.main.dynamic_workflow"},"level":"error","msg":"failed Execute for node. Error: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]","ts":"2023-02-08T12:50:04Z"}
[Feb 8, 2023 6:20:04 PM GMT+5]
{"json":{"exec_id":"ahlrb9l95q2h4gg8fhgf","node":"n0","ns":"development","res_ver":"55914809","routine":"worker-14","wf":"flyteexamples:development:dynamic_workflow.main.dynamic_workflow"},"level":"error","msg":"handling dynamic subnodes failed with error: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]","ts":"2023-02-08T12:50:04Z"}
[Feb 8, 2023 6:20:04 PM GMT+5]
{"json":{"exec_id":"ahlrb9l95q2h4gg8fhgf","node":"n0","ns":"development","res_ver":"55914809","routine":"worker-14","wf":"flyteexamples:development:dynamic_workflow.main.dynamic_workflow"},"level":"error","msg":"failed Execute for node. Error: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]","ts":"2023-02-08T12:50:04Z"}
[Feb 8, 2023 6:20:04 PM GMT+5]
{"json":{"exec_id":"ahlrb9l95q2h4gg8fhgf","ns":"development","res_ver":"55914809","routine":"worker-14","wf":"flyteexamples:development:dynamic_workflow.main.dynamic_workflow"},"level":"warning","msg":"Error in handling running workflow [InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]]","ts":"2023-02-08T12:50:04Z"}
[Feb 8, 2023 6:20:04 PM GMT+5]
{"json":{"exec_id":"ahlrb9l95q2h4gg8fhgf","ns":"development","res_ver":"55914809","routine":"worker-14","wf":"flyteexamples:development:dynamic_workflow.main.dynamic_workflow"},"level":"error","msg":"Error when trying to reconcile workflow. Error [InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]]. Error Type[*errors.EventError]","ts":"2023-02-08T12:50:04Z"}
[Feb 8, 2023 6:20:04 PM GMT+5]
E0208 12:50:04.456196 1 workers.go:102] error syncing 'development/ahlrb9l95q2h4gg8fhgf': InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
Bernhard Stadlbauer
02/08/2023, 12:53 PMHIMANSHU JOSHI
02/08/2023, 12:59 PMBernhard Stadlbauer
02/08/2023, 1:09 PMHIMANSHU JOSHI
02/08/2023, 3:00 PMKetan (kumare3)
HIMANSHU JOSHI
02/08/2023, 3:01 PMKetan (kumare3)
HIMANSHU JOSHI
02/08/2023, 4:18 PM{"json":{"exec_id":"as6j9bshk8qx5rfc6x7f","node":"n0","ns":"development","res_ver":"56079374","routine":"worker-0","wf":"flyteexamples:development:dynamic_workflow.main.dynamic_workflow"},"level":"error","msg":"failed Execute for node. Error: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]","ts":"2023-02-08T16:14:31Z"}
{"json":{"exec_id":"as6j9bshk8qx5rfc6x7f","ns":"development","res_ver":"56079374","routine":"worker-0","wf":"flyteexamples:development:dynamic_workflow.main.dynamic_workflow"},"level":"warning","msg":"Error in handling running workflow [InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]]","ts":"2023-02-08T16:14:31Z"}
@Ketan (kumare3) @Bernhard Stadlbauer still the same error even after downgrading flytekit to 1.2.7. do i also need to downgrade flytepropeller / <all the deployments> docker image to the same version?Eduardo Apolinario (eapolinario)
02/08/2023, 4:23 PMHIMANSHU JOSHI
02/08/2023, 4:25 PMEduardo Apolinario (eapolinario)
02/08/2023, 4:26 PMHIMANSHU JOSHI
02/08/2023, 4:26 PMpyflyte --pkgs dynamic_workflow.main package --image <image-name> -f -o dynamic_workflow.tgz
flytectl register files --project flyteexamples --domain development --archive dynamic_workflow.tgz --version v2
Eduardo Apolinario (eapolinario)
02/08/2023, 4:31 PMHIMANSHU JOSHI
02/08/2023, 4:33 PMEduardo Apolinario (eapolinario)
02/08/2023, 11:42 PMKetan (kumare3)
HIMANSHU JOSHI
02/09/2023, 5:55 AMgrpc
connection through pyflyte
?Eduardo Apolinario (eapolinario)
02/09/2023, 6:57 AMinsecure: true
in the config file. What error are you seeing?HIMANSHU JOSHI
02/09/2023, 1:27 PMgrpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses; last error: INTERNAL: Trying to connect an http1.x server"
debug_error_string = "UNKNOWN:Failed to pick subchannel {created_time:"2023-02-09T18:57:01.276539+05:30", children:[UNKNOWN:failed to connect to all addresses; last error: INTERNAL: Trying to connect an http1.x server {grpc_status:14, created_time:"2023-02-09T18:57:01.27653+05:30"}]}"
sorry just saw the message this is the error i'm getting for pyflyte
if i put insecure:true
and for flytectl
,
rpc error: code = Unavailable desc = connection closed before server preface received
but error with flytectl
goes away if i use insecure: false
and insecureSkipVerify: true
but than i get openssl handshake failed
for pyflyte
in that case
P.s TLS
disabled
Hope I was able to explain the problem😅Ketan (kumare3)