abundant-hamburger-66584
06/14/2022, 8:52 PM--remote
command. I am running into an issue. I am not sure where to start looking, so any advice is much appreciated.
Inside of a new venv
$git clone <https://github.com/flyteorg/flytesnacks>
$cd flytesnacks/cookbook
$pip install -r core/requirements.txt
$flytectl demo start
$pyflyte run --remote core/flyte_basics/hello_world.py:my_wf
And then I get this
Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses"
debug_error_string = "{"created":"@1655239314.572674000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3129,"referenced_errors":[{"created":"@1655239314.572673000","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}"
freezing-airport-6809
freezing-airport-6809
high-accountant-32689
06/15/2022, 1:31 AMpyflyte
command with more verbose logs (run FLYTE_SDK_LOGGING_LEVEL=20 pyflyte run --remote core/flyte_basics/hello_world.py:my_wf
). These logs will tell you which config file was used.
3. Can you confirm the contents of the config file found in the previous step?adamant-zoo-87864
06/15/2022, 10:21 AM➜ cookbook git:(master) flytectl version
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags.
{
"App": "flytectl",
"Build": "e92f7fc",
"Version": "0.6.0",
"BuildTime": "2022-06-15 17:22:44.272166 +0700 +07 m=+0.061320352"
}%
2. re-run the pyflyte
command with more verbose logs
➜ cookbook git:(master) FLYTE_SDK_LOGGING_LEVEL=20 pyflyte run --remote core/flyte_basics/hello_world.py:my_wf
{"asctime": "2022-06-15 17:18:09,655", "name": "flytekit", "levelname": "INFO", "message": "Setting protocol to file"}
{"asctime": "2022-06-15 17:18:10,462", "name": "flytekit", "levelname": "INFO", "message": "Setting protocol to file"}
{"asctime": "2022-06-15 17:18:10,463", "name": "flytekit", "levelname": "INFO", "message": "Setting protocol to file"}
{"asctime": "2022-06-15 17:18:10,463", "name": "flytekit", "levelname": "INFO", "message": "Setting protocol to file"}
{"asctime": "2022-06-15 17:18:10,465", "name": "flytekit", "levelname": "INFO", "message": "We won't register bigquery handler for structured dataset because we can't find the packages google-cloud-bigquery-storage and google-cloud-bigquery"}
E0615 17:18:10.556420000 4376362496 <http://ssl_transport_security.cc:1504]|ssl_transport_security.cc:1504]> Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
E0615 17:18:10.558737000 4376362496 <http://ssl_transport_security.cc:1504]|ssl_transport_security.cc:1504]> Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
{"asctime": "2022-06-15 17:18:10,559", "name": "flytekit.cli", "levelname": "INFO", "message": "Flyte Client configured -> localhost:30081 in secure mode."}
{"asctime": "2022-06-15 17:18:10,560", "name": "flytekit", "levelname": "INFO", "message": "Setting protocol to file"}
{"asctime": "2022-06-15 17:18:10,567", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1655288290.567373000\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3129,\"referenced_errors\":[{\"created\":\"@1655288290.567372000\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":163,\"grpc_status\":14}]}\"\n>, sleeping 200ms and retrying"}
{"asctime": "2022-06-15 17:18:10,771", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1655288290.770694000\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3129,\"referenced_errors\":[{\"created\":\"@1655288290.770693000\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":163,\"grpc_status\":14}]}\"\n>, sleeping 400ms and retrying"}
Traceback (most recent call last):
File "/usr/local/bin/pyflyte", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/flytekit/clis/sdk_in_container/run.py", line 152, in run
wf = remote.register_script(
File "/usr/local/lib/python3.9/site-packages/flytekit/remote/remote.py", line 517, in register_script
upload_location, md5_version = fast_register_single_script(
File "/usr/local/lib/python3.9/site-packages/flytekit/tools/script_mode.py", line 118, in fast_register_single_script
upload_location = create_upload_location_fn(content_md5=md5)
File "/usr/local/lib/python3.9/site-packages/flytekit/clients/friendly.py", line 998, in get_upload_signed_url
return super(SynchronousFlyteClient, self).create_upload_location(
File "/usr/local/lib/python3.9/site-packages/flytekit/clients/raw.py", line 40, in handler
return fn(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/flytekit/clients/raw.py", line 834, in create_upload_location
return self._dataproxy_stub.CreateUploadLocation(create_upload_location_request, metadata=self._metadata)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses"
debug_error_string = "{"created":"@1655288291.172324000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3129,"referenced_errors":[{"created":"@1655288291.172323000","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}"
>
freezing-airport-6809
KUBECONFIG=$KUBECONFIG:/Users/evan.sadler/.kube/config:/Users/evan.sadler/.flyte/k3s/k3s.yaml
export FLYTECTL_CONFIG=/Users/evan.sadler/.flyte/config-sandbox.yaml
abundant-hamburger-66584
06/15/2022, 1:30 PMbroad-monitor-993
06/15/2022, 1:44 PMbroad-monitor-993
06/15/2022, 1:49 PMNote
admonitionabundant-hamburger-66584
06/15/2022, 1:50 PMbroad-monitor-993
06/15/2022, 1:50 PMflytectl demo start
I’m not seeing the export statements in my stdout… did we remove them, or did we just forget to add it?? @freezing-airport-6809 @icy-agent-73298 @great-school-54368
👨💻 Flyte is ready! Flyte UI is available at <http://localhost:30080/console> 🚀 🚀 🎉
However with flytectl sandbox start
I see the following:
👨💻 Flyte is ready! Flyte UI is available at <http://localhost:30081/console> 🚀 🚀 🎉
Add KUBECONFIG and FLYTECTL_CONFIG to your environment variable
export KUBECONFIG=$KUBECONFIG:/Users/nielsbantilan/.kube/config:/Users/nielsbantilan/.flyte/k3s/k3s.yaml
export FLYTECTL_CONFIG=/Users/nielsbantilan/.flyte/config-sandbox.yaml
We should add those export statements to flytectl demo start
abundant-hamburger-66584
06/15/2022, 1:51 PMabundant-hamburger-66584
06/15/2022, 1:51 PMbroad-monitor-993
06/15/2022, 1:52 PMsandbox
and demo
are effectively the same for the most part, but demo
starts up way faster.abundant-hamburger-66584
06/15/2022, 1:57 PMFile "/Users/evan.sadler/miniconda3/envs/hbo/lib/python3.9/site-packages/flytekit/clients/raw.py", line 55, in handler
raise _user_exceptions.FlyteEntityNotExistException(e)
flytekit.exceptions.user.FlyteEntityNotExistException: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.NOT_FOUND
details = "missing entity of type WORKFLOW with identifier project:"flytesnacks" domain:"development" name:"flyte.workflows.retention_example.run_all_model_configs" version:"6ik5rx9XN7kPjwiZU3xdxg==" "
debug_error_string = "{"created":"@1655301388.336135000","description":"Error received from peer ipv6:[::1]:30081","file":"src/core/lib/surface/call.cc","file_line":904,"grpc_message":"missing entity of type WORKFLOW with identifier project:"flytesnacks" domain:"development" name:"flyte.workflows.retention_example.run_all_model_configs" version:"6ik5rx9XN7kPjwiZU3xdxg==" ","grpc_status":5}"
broad-monitor-993
06/15/2022, 1:58 PMabundant-hamburger-66584
06/15/2022, 1:58 PMicy-agent-73298
06/15/2022, 2:24 PMflytectl register files
meanwhile https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_register_files.htmlabundant-hamburger-66584
06/15/2022, 2:54 PM$ pyflyte --pkgs flyte.workflows serialize --local-source-root ./ --image retention:64c7ae739661a211efcfc0f977e56a4c782893a7 workflows -f _pb_output/
$flytectl register files _pb_output/* -p flytesnacks -d development --version v1
--------------------------------------------------------------------------- --------- ------------------------------
| NAME (6) | STATUS | ADDITIONAL INFO |
--------------------------------------------------------------------------- --------- ------------------------------
| _pb_output/0_components.example.snowflake_unload_1.pb | Success | Successfully registered file |
--------------------------------------------------------------------------- --------- ------------------------------
| _pb_output/1_components.example.preprocess_1.pb | Success | Successfully registered file |
--------------------------------------------------------------------------- --------- ------------------------------
| _pb_output/2_components.example.lightgbm_train_1.pb | Success | Successfully registered file |
--------------------------------------------------------------------------- --------- ------------------------------
| _pb_output/3_components.example.classifier_validation_1.pb | Success | Successfully registered file |
--------------------------------------------------------------------------- --------- ------------------------------
| _pb_output/4_flyte.workflows.retention_example.run_config_1.pb | Success | Successfully registered file |
--------------------------------------------------------------------------- --------- ------------------------------
| _pb_output/5_flyte.workflows.retention_example.run_all_model_configs_1.pb | Success | Successfully registered file |
--------------------------------------------------------------------------- --------- ------------------------------
abundant-hamburger-66584
06/15/2022, 2:55 PMfreezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
broad-monitor-993
06/15/2022, 3:02 PMabundant-hamburger-66584
06/15/2022, 4:24 PMabundant-hamburger-66584
06/15/2022, 4:28 PM./docker_build_and_tag.sh
. I want all my components to use it by default. Do I need to use the --image
tag to use it?
pyflyte run --remote --image retention:64c7ae739661a211efcfc0f977e56a4c782893a7 flyte/workflows/retention_example.py run_all_model_configs_wf
abundant-hamburger-66584
06/15/2022, 4:30 PM--image
I get this
[1/1] currentAttempt done. Last Error: USER::containers with unready status: [f5891b9ba9721454887a-n0-0]|Back-off pulling image "retention:64c7ae739661a211efcfc0f977e56a4c782893a7"
abundant-hamburger-66584
06/15/2022, 4:30 PMabundant-hamburger-66584
06/15/2022, 4:44 PMdocker run -d -p 5000:5000 --restart=always --name local-registry registry:2
high-accountant-32689
06/15/2022, 4:50 PMflytectl
and build your image there (in order to exec into the right container do flytectl sandbox exec -- <command>
)high-accountant-32689
06/15/2022, 4:50 PM--image
in the call to pyflyte run
as you are doingabundant-hamburger-66584
06/15/2022, 4:52 PMflytectl sandbox exec --
. Would you mind providing more information?high-accountant-32689
06/15/2022, 4:59 PMflytectl sandbox exec -- ./docker_build_and_tag.sh
, assuming that the script is present in the dir you used to start your sandbox.high-accountant-32689
06/15/2022, 4:59 PMflytectl demo start
, then you will be able to do flytectl demo exec -- <command>
)abundant-hamburger-66584
06/15/2022, 5:03 PMabundant-hamburger-66584
06/16/2022, 12:34 AMworried-restaurant-93221
07/11/2022, 11:54 AMpyflyte run --remote
and get this Error:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNIMPLEMENTED
details = "Received http2 header with status: 404"
debug_error_string = "{"created":"@1657540045.111245779","description":"Error received from peer ipv4:10.194.0.6:443","file":"src/core/lib/surface/call.cc","file_line":952,"grpc_message":"Received http2 header with status: 404","grpc_status":12}"
We're running flyteadmin 1.1.25 currently - it should already implement the CreateUploadLocationRequest
where this failsfreezing-airport-6809
icy-agent-73298
07/11/2022, 2:40 PMworried-restaurant-93221
07/12/2022, 7:40 AMworried-restaurant-93221
07/12/2022, 3:14 PMicy-agent-73298
07/12/2022, 3:45 PMfreezing-airport-6809
freezing-boots-56761