echoing-river-3611
02/07/2023, 1:28 PMgrpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.25.0.46:443: Ssl handshake failed: SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED"
debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.25.0.46:443: Ssl handshake failed: SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED {grpc_status:14, created_time:"2023-02-07T18:47:06.951145+05:30"}"
Any body has an idea what is the problem here?
P.s I have TLS disabled on ingress.agreeable-kitchen-44189
02/07/2023, 1:39 PMadmin.insecure: True
set in your ~/.flyte/config.yaml
?echoing-river-3611
02/07/2023, 1:41 PMechoing-river-3611
02/07/2023, 1:43 PMadmin:
endpoint: "dns:///"
# authType: Pkce
insecure: false
insecureSkipVerify: true
console:
endpoint: ""
logger:
show-source: true
level: 0
echoing-river-3611
02/07/2023, 1:44 PMflytectl get projects
echoing-river-3611
02/07/2023, 1:49 PMagreeable-kitchen-44189
02/07/2023, 1:55 PMechoing-river-3611
02/07/2023, 2:27 PMaverage-finland-92144
02/07/2023, 3:10 PMaverage-finland-92144
02/07/2023, 3:22 PMechoing-river-3611
02/07/2023, 4:09 PMfreezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
echoing-river-3611
02/07/2023, 4:30 PMName: grpcio
Version: 1.51.1
Summary: HTTP/2-based RPC framework
Home-page: <https://grpc.io>
Author: The gRPC Authors
Author-email: <mailto:grpc-io@googlegroups.com|grpc-io@googlegroups.com>
License: Apache License 2.0
Location: "<package-path>"
Requires:
Required-by: flytekit, grpcio-status
echoing-river-3611
02/07/2023, 4:31 PMhigh-accountant-32689
02/07/2023, 4:45 PMstocky-notebook-88311
02/07/2023, 4:49 PMstocky-notebook-88311
02/07/2023, 4:49 PMechoing-river-3611
02/07/2023, 5:04 PMfuture-monitor-58430
02/07/2023, 5:17 PMstocky-notebook-88311
02/07/2023, 5:17 PMechoing-river-3611
02/07/2023, 5:18 PMfuture-monitor-58430
02/07/2023, 5:19 PMhttps
, can you click on the not secure
future-monitor-58430
02/07/2023, 5:19 PMcertificate is not valid
future-monitor-58430
02/07/2023, 5:19 PMechoing-river-3611
02/07/2023, 5:19 PMfuture-monitor-58430
02/07/2023, 5:20 PMechoing-river-3611
02/07/2023, 5:20 PMfuture-monitor-58430
02/07/2023, 5:21 PMIssue by
-> Common Name (CN)
say?echoing-river-3611
02/07/2023, 5:21 PMfuture-monitor-58430
02/07/2023, 5:22 PMfuture-monitor-58430
02/07/2023, 5:22 PMechoing-river-3611
02/07/2023, 5:23 PMfuture-monitor-58430
02/07/2023, 5:23 PMechoing-river-3611
02/07/2023, 5:24 PMechoing-river-3611
02/07/2023, 5:24 PMechoing-river-3611
02/07/2023, 5:24 PMfuture-monitor-58430
02/07/2023, 5:26 PMadmin:
caCertFilePath: /path/to/ca-bundle.crt
echoing-river-3611
02/07/2023, 5:27 PMechoing-river-3611
02/07/2023, 5:27 PMechoing-river-3611
02/07/2023, 5:27 PMfuture-monitor-58430
02/07/2023, 5:27 PMfuture-monitor-58430
02/07/2023, 5:28 PMechoing-river-3611
02/07/2023, 5:30 PMfuture-monitor-58430
02/07/2023, 5:31 PMechoing-river-3611
02/07/2023, 5:33 PMechoing-river-3611
02/07/2023, 5:33 PMfuture-monitor-58430
02/07/2023, 5:33 PMfuture-monitor-58430
02/07/2023, 5:34 PMfuture-monitor-58430
02/07/2023, 5:35 PMadmin.insecure: True
echoing-river-3611
02/08/2023, 12:18 PMpyflyte package
i tried it and it throws an error i.e. submodule not found?
P.s We're still not able to resolve ssl issue so can't use pyflyte register bcz of thattall-lock-23197
pyflyte package
should work. Could you share the exact error?echoing-river-3611
02/08/2023, 12:20 PMechoing-river-3611
02/08/2023, 12:20 PMtall-lock-23197
pyflyte package
contain utils
?echoing-river-3611
02/08/2023, 12:22 PMpyflyte --pkgs dynamic_workflow package --image <image-name> -f -o dynamic_workflow.tgz
command for referenceechoing-river-3611
02/08/2023, 12:22 PMDoes the package you're sending toyescontainpyflyte package
?utils
echoing-river-3611
02/08/2023, 12:24 PMtall-lock-23197
from .utils ...
echoing-river-3611
02/08/2023, 12:50 PMError when trying to reconcile workflow
and in the dynamic task it was suppose to create 5 different task as i have given 5 input it did create those but it's been 20 min it is still stuck in queued and propeller is throwing above mentioned error and only one pod for dynamic task was created expecting 5agreeable-kitchen-44189
02/08/2023, 12:52 PMrpc error: code = InvalidArgument desc = missing project
somewhere in your propeller logs?echoing-river-3611
02/08/2023, 12:52 PMechoing-river-3611
02/08/2023, 12:52 PM[Feb 8, 2023 6:20:04 PM GMT+5]
{"json":{"exec_id":"ahlrb9l95q2h4gg8fhgf","node":"n0/dn0","ns":"development","res_ver":"55914809","routine":"worker-14","wf":"flyteexamples:development:dynamic_workflow.main.dynamic_workflow"},"level":"error","msg":"handling parent node failed with error: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]","ts":"2023-02-08T12:50:04Z"}
[Feb 8, 2023 6:20:04 PM GMT+5]
{"json":{"exec_id":"ahlrb9l95q2h4gg8fhgf","node":"n0/dn0","ns":"development","res_ver":"55914809","routine":"worker-14","wf":"flyteexamples:development:dynamic_workflow.main.dynamic_workflow"},"level":"error","msg":"failed Execute for node. Error: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]","ts":"2023-02-08T12:50:04Z"}
[Feb 8, 2023 6:20:04 PM GMT+5]
{"json":{"exec_id":"ahlrb9l95q2h4gg8fhgf","node":"n0","ns":"development","res_ver":"55914809","routine":"worker-14","wf":"flyteexamples:development:dynamic_workflow.main.dynamic_workflow"},"level":"error","msg":"handling dynamic subnodes failed with error: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]","ts":"2023-02-08T12:50:04Z"}
[Feb 8, 2023 6:20:04 PM GMT+5]
{"json":{"exec_id":"ahlrb9l95q2h4gg8fhgf","node":"n0","ns":"development","res_ver":"55914809","routine":"worker-14","wf":"flyteexamples:development:dynamic_workflow.main.dynamic_workflow"},"level":"error","msg":"failed Execute for node. Error: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]","ts":"2023-02-08T12:50:04Z"}
[Feb 8, 2023 6:20:04 PM GMT+5]
{"json":{"exec_id":"ahlrb9l95q2h4gg8fhgf","ns":"development","res_ver":"55914809","routine":"worker-14","wf":"flyteexamples:development:dynamic_workflow.main.dynamic_workflow"},"level":"warning","msg":"Error in handling running workflow [InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]]","ts":"2023-02-08T12:50:04Z"}
[Feb 8, 2023 6:20:04 PM GMT+5]
{"json":{"exec_id":"ahlrb9l95q2h4gg8fhgf","ns":"development","res_ver":"55914809","routine":"worker-14","wf":"flyteexamples:development:dynamic_workflow.main.dynamic_workflow"},"level":"error","msg":"Error when trying to reconcile workflow. Error [InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]]. Error Type[*errors.EventError]","ts":"2023-02-08T12:50:04Z"}
[Feb 8, 2023 6:20:04 PM GMT+5]
E0208 12:50:04.456196 1 workers.go:102] error syncing 'development/ahlrb9l95q2h4gg8fhgf': InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
echoing-river-3611
02/08/2023, 12:52 PMagreeable-kitchen-44189
02/08/2023, 12:53 PMagreeable-kitchen-44189
02/08/2023, 12:53 PMechoing-river-3611
02/08/2023, 12:59 PMechoing-river-3611
02/08/2023, 12:59 PMagreeable-kitchen-44189
02/08/2023, 1:09 PMagreeable-kitchen-44189
02/08/2023, 1:53 PMechoing-river-3611
02/08/2023, 3:00 PMfreezing-airport-6809
freezing-airport-6809
echoing-river-3611
02/08/2023, 3:01 PMfreezing-airport-6809
freezing-airport-6809
echoing-river-3611
02/08/2023, 4:18 PM{"json":{"exec_id":"as6j9bshk8qx5rfc6x7f","node":"n0","ns":"development","res_ver":"56079374","routine":"worker-0","wf":"flyteexamples:development:dynamic_workflow.main.dynamic_workflow"},"level":"error","msg":"failed Execute for node. Error: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]","ts":"2023-02-08T16:14:31Z"}
{"json":{"exec_id":"as6j9bshk8qx5rfc6x7f","ns":"development","res_ver":"56079374","routine":"worker-0","wf":"flyteexamples:development:dynamic_workflow.main.dynamic_workflow"},"level":"warning","msg":"Error in handling running workflow [InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]]","ts":"2023-02-08T16:14:31Z"}
@freezing-airport-6809 @agreeable-kitchen-44189 still the same error even after downgrading flytekit to 1.2.7. do i also need to downgrade flytepropeller / <all the deployments> docker image to the same version?high-accountant-32689
02/08/2023, 4:23 PMechoing-river-3611
02/08/2023, 4:25 PMhigh-accountant-32689
02/08/2023, 4:26 PMechoing-river-3611
02/08/2023, 4:26 PMpyflyte --pkgs dynamic_workflow.main package --image <image-name> -f -o dynamic_workflow.tgz
flytectl register files --project flyteexamples --domain development --archive dynamic_workflow.tgz --version v2
echoing-river-3611
02/08/2023, 4:29 PMhigh-accountant-32689
02/08/2023, 4:31 PMechoing-river-3611
02/08/2023, 4:33 PMechoing-river-3611
02/08/2023, 4:54 PMhigh-accountant-32689
02/08/2023, 11:42 PMfreezing-airport-6809
echoing-river-3611
02/09/2023, 5:55 AMgrpc
connection through pyflyte
?high-accountant-32689
02/09/2023, 6:57 AMinsecure: true
in the config file. What error are you seeing?echoing-river-3611
02/09/2023, 1:27 PMgrpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses; last error: INTERNAL: Trying to connect an http1.x server"
debug_error_string = "UNKNOWN:Failed to pick subchannel {created_time:"2023-02-09T18:57:01.276539+05:30", children:[UNKNOWN:failed to connect to all addresses; last error: INTERNAL: Trying to connect an http1.x server {grpc_status:14, created_time:"2023-02-09T18:57:01.27653+05:30"}]}"
sorry just saw the message this is the error i'm getting for pyflyte
if i put insecure:true
and for flytectl
,
rpc error: code = Unavailable desc = connection closed before server preface received
but error with flytectl
goes away if i use insecure: false
and insecureSkipVerify: true
but than i get openssl handshake failed
for pyflyte
in that case
P.s TLS
disabled
Hope I was able to explain the problem😅echoing-river-3611
02/09/2023, 5:02 PMfreezing-airport-6809
freezing-airport-6809