I am trying to use the flyte demo locally (OSX intel chip) using the

--remote

command. I am running into an issue. I am not sure where to start looking, so any advice is much appreciated. Inside of a new venv

$git clone <https://github.com/flyteorg/flytesnacks>
$cd flytesnacks/cookbook
$pip install -r core/requirements.txt

$flytectl demo start
$pyflyte run --remote core/flyte_basics/hello_world.py:my_wf

And then I get this

Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1655239314.572674000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3129,"referenced_errors":[{"created":"@1655239314.572673000","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}"

hmm weird

@Evan Sadler, you shouldn't be seeing SSL errors in this case. A few things to double-check: 1. can you double-check which version of flytectl you're running? 2. Can you re-run the

pyflyte

command with more verbose logs (run

FLYTE_SDK_LOGGING_LEVEL=20 pyflyte run --remote core/flyte_basics/hello_world.py:my_wf

). These logs will tell you which config file was used. 3. Can you confirm the contents of the config file found in the previous step?

I have the same issue. 1. The Flytectl version that I have installed is 0.6.0

➜  cookbook git:(master) flytectl version
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
{
  "App": "flytectl",
  "Build": "e92f7fc",
  "Version": "0.6.0",
  "BuildTime": "2022-06-15 17:22:44.272166 +0700 +07 m=+0.061320352"
}%

2. re-run the

pyflyte

command with more verbose logs

➜  cookbook git:(master) FLYTE_SDK_LOGGING_LEVEL=20 pyflyte run --remote core/flyte_basics/hello_world.py:my_wf
{"asctime": "2022-06-15 17:18:09,655", "name": "flytekit", "levelname": "INFO", "message": "Setting protocol to file"}
{"asctime": "2022-06-15 17:18:10,462", "name": "flytekit", "levelname": "INFO", "message": "Setting protocol to file"}
{"asctime": "2022-06-15 17:18:10,463", "name": "flytekit", "levelname": "INFO", "message": "Setting protocol to file"}
{"asctime": "2022-06-15 17:18:10,463", "name": "flytekit", "levelname": "INFO", "message": "Setting protocol to file"}
{"asctime": "2022-06-15 17:18:10,465", "name": "flytekit", "levelname": "INFO", "message": "We won't register bigquery handler for structured dataset because we can't find the packages google-cloud-bigquery-storage and google-cloud-bigquery"}
E0615 17:18:10.556420000 4376362496 <http://ssl_transport_security.cc:1504]|ssl_transport_security.cc:1504]>    Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
E0615 17:18:10.558737000 4376362496 <http://ssl_transport_security.cc:1504]|ssl_transport_security.cc:1504]>    Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
{"asctime": "2022-06-15 17:18:10,559", "name": "flytekit.cli", "levelname": "INFO", "message": "Flyte Client configured -> localhost:30081 in secure mode."}
{"asctime": "2022-06-15 17:18:10,560", "name": "flytekit", "levelname": "INFO", "message": "Setting protocol to file"}
{"asctime": "2022-06-15 17:18:10,567", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1655288290.567373000\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3129,\"referenced_errors\":[{\"created\":\"@1655288290.567372000\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":163,\"grpc_status\":14}]}\"\n>, sleeping 200ms and retrying"}
{"asctime": "2022-06-15 17:18:10,771", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1655288290.770694000\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3129,\"referenced_errors\":[{\"created\":\"@1655288290.770693000\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":163,\"grpc_status\":14}]}\"\n>, sleeping 400ms and retrying"}
Traceback (most recent call last):
  File "/usr/local/bin/pyflyte", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/flytekit/clis/sdk_in_container/run.py", line 152, in run
    wf = remote.register_script(
  File "/usr/local/lib/python3.9/site-packages/flytekit/remote/remote.py", line 517, in register_script
    upload_location, md5_version = fast_register_single_script(
  File "/usr/local/lib/python3.9/site-packages/flytekit/tools/script_mode.py", line 118, in fast_register_single_script
    upload_location = create_upload_location_fn(content_md5=md5)
  File "/usr/local/lib/python3.9/site-packages/flytekit/clients/friendly.py", line 998, in get_upload_signed_url
    return super(SynchronousFlyteClient, self).create_upload_location(
  File "/usr/local/lib/python3.9/site-packages/flytekit/clients/raw.py", line 40, in handler
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/flytekit/clients/raw.py", line 834, in create_upload_location
    return self._dataproxy_stub.CreateUploadLocation(create_upload_location_request, metadata=self._metadata)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1655288291.172324000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3129,"referenced_errors":[{"created":"@1655288291.172323000","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}"
>

@Oliver Nguyen - @Evan Sadler got it running. He missed a step after flytectl demo start - Okay it works now. Needed to add these…somehow I missed them!

KUBECONFIG=$KUBECONFIG:/Users/evan.sadler/.kube/config:/Users/evan.sadler/.flyte/k3s/k3s.yaml
export FLYTECTL_CONFIG=/Users/evan.sadler/.flyte/config-sandbox.yaml

Thanks for the help!

I think it some point we deleted the lines that explicitly point to those export statements:

Cool. Are you sure they print out every time?

after running

flytectl demo start

I’m not seeing the export statements in my stdout… did we remove them, or did we just forget to add it?? @Ketan (kumare3) @Prafulla Mahindrakar @Yuvraj

👨‍💻 Flyte is ready! Flyte UI is available at <http://localhost:30080/console> 🚀 🚀 🎉

However with

flytectl sandbox start

I see the following:

👨‍💻 Flyte is ready! Flyte UI is available at <http://localhost:30081/console> 🚀 🚀 🎉
Add KUBECONFIG and FLYTECTL_CONFIG to your environment variable
export KUBECONFIG=$KUBECONFIG:/Users/nielsbantilan/.kube/config:/Users/nielsbantilan/.flyte/k3s/k3s.yaml
export FLYTECTL_CONFIG=/Users/nielsbantilan/.flyte/config-sandbox.yaml

We should add those export statements to

flytectl demo start

This is exactly what happened…I tried the sandbox start

cool, yeah so

sandbox

and

demo

are effectively the same for the most part, but

demo

starts up way faster.

Sweet. @Niels Bantilan I am struggling to deploy my workflow to the local flyte sanbox. Would you be able to help me quickly debug? I can get the hello world demo working, but when I run my own workflow it fails with this:

File "/Users/evan.sadler/miniconda3/envs/hbo/lib/python3.9/site-packages/flytekit/clients/raw.py", line 55, in handler
    raise _user_exceptions.FlyteEntityNotExistException(e)
flytekit.exceptions.user.FlyteEntityNotExistException: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.NOT_FOUND
	details = "missing entity of type WORKFLOW with identifier project:"flytesnacks" domain:"development" name:"flyte.workflows.retention_example.run_all_model_configs" version:"6ik5rx9XN7kPjwiZU3xdxg==" "
	debug_error_string = "{"created":"@1655301388.336135000","description":"Error received from peer ipv6:[::1]:30081","file":"src/core/lib/surface/call.cc","file_line":904,"grpc_message":"missing entity of type WORKFLOW with identifier project:"flytesnacks" domain:"development" name:"flyte.workflows.retention_example.run_all_model_configs" version:"6ik5rx9XN7kPjwiZU3xdxg==" ","grpc_status":5}"

sure, I need to step out for an hour, will be back to help debug then!

Thanks!

@Niels Bantilan it seems we didn’t have those exports. will add those . @Evan Sadler seems the registration step wasnt done. Can you try registering your workflow using

flytectl register  files

meanwhile https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_register_files.html

@Prafulla Mahindrakar I did this and I am getting the same error. THe dynamic workflows look like they are being registered as tasks in the UI. Any ideas?

$ pyflyte --pkgs flyte.workflows serialize --local-source-root ./ --image retention:64c7ae739661a211efcfc0f977e56a4c782893a7 workflows -f _pb_output/
$flytectl register files _pb_output/* -p flytesnacks -d development --version v1

--------------------------------------------------------------------------- --------- ------------------------------
| NAME (6)                                                                  | STATUS  | ADDITIONAL INFO              |
 --------------------------------------------------------------------------- --------- ------------------------------
| _pb_output/0_components.example.snowflake_unload_1.pb                     | Success | Successfully registered file |
 --------------------------------------------------------------------------- --------- ------------------------------
| _pb_output/1_components.example.preprocess_1.pb                           | Success | Successfully registered file |
 --------------------------------------------------------------------------- --------- ------------------------------
| _pb_output/2_components.example.lightgbm_train_1.pb                       | Success | Successfully registered file |
 --------------------------------------------------------------------------- --------- ------------------------------
| _pb_output/3_components.example.classifier_validation_1.pb                | Success | Successfully registered file |
 --------------------------------------------------------------------------- --------- ------------------------------
| _pb_output/4_flyte.workflows.retention_example.run_config_1.pb            | Success | Successfully registered file |
 --------------------------------------------------------------------------- --------- ------------------------------
| _pb_output/5_flyte.workflows.retention_example.run_all_model_configs_1.pb | Success | Successfully registered file |
 --------------------------------------------------------------------------- --------- ------------------------------

@dynamic is a task

yep, we call them “dynamic workflows” because the end state is a workflow, but it uses tasks under the hood to dynamically generate the workflow at runtime.

Sweet! Thanks okay I see the workflow

@Evan Sadler, both the demo and the sandbox clusters run inside a docker container, so you have to build your image using the docker daemon that's running inside this outer docker container. You can exec to the right image via

flytectl

and build your image there (in order to exec into the right container do

flytectl sandbox exec -- <command>

)

Thank you! I am confused on how to use

flytectl sandbox exec --

. Would you mind providing more information?

sure, maybe an image will help: https://docs.flyte.org/en/latest/deployment/sandbox.html#flyte-sandbox-as-a-single-docker-container. So in your case, you should be able to just do

flytectl sandbox exec -- ./docker_build_and_tag.sh

, assuming that the script is present in the dir you used to start your sandbox.

Oh great. I will restart the the sandbox and try this!

I also wanted to finally try out the new

pyflyte run --remote

and get this Error:

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNIMPLEMENTED
        details = "Received http2 header with status: 404"
        debug_error_string = "{"created":"@1657540045.111245779","description":"Error received from peer ipv4:10.194.0.6:443","file":"src/core/lib/surface/call.cc","file_line":952,"grpc_message":"Received http2 header with status: 404","grpc_status":12}"

We're running flyteadmin 1.1.25 currently - it should already implement the

CreateUploadLocationRequest

where this fails

Ingress?

Hi @Tim Bauer can you check your ingress config for DataProxy specifically these https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/templates/common/ingress.yaml#L27-L50

Ah yes we have an older manifest missing those paths, thank you both! 🙏

Its been hard maintaining both kustomize and helm and seems that might be the reason you are seeing those differences. Helm is the recommended way we are asking users to manage there flyte deployments.

Cc @jeev at some point freenome had volunteered to maintain kustomize

we’re happy to at least while we’re using kustomize internally