https://flyte.org logo
e

Evan Sadler

06/14/2022, 8:52 PM
I am trying to use the flyte demo locally (OSX intel chip) using the
--remote
command. I am running into an issue. I am not sure where to start looking, so any advice is much appreciated. Inside of a new venv
Copy code
$git clone <https://github.com/flyteorg/flytesnacks>
$cd flytesnacks/cookbook
$pip install -r core/requirements.txt

$flytectl demo start
$pyflyte run --remote core/flyte_basics/hello_world.py:my_wf
And then I get this
Copy code
Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1655239314.572674000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3129,"referenced_errors":[{"created":"@1655239314.572673000","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}"
k

Ketan (kumare3)

06/14/2022, 8:53 PM
hmm weird
there is no SSL
e

Eduardo Apolinario (eapolinario)

06/15/2022, 1:31 AM
@Evan Sadler, you shouldn't be seeing SSL errors in this case. A few things to double-check: 1. can you double-check which version of flytectl you're running? 2. Can you re-run the
pyflyte
command with more verbose logs (run
FLYTE_SDK_LOGGING_LEVEL=20 pyflyte run --remote core/flyte_basics/hello_world.py:my_wf
). These logs will tell you which config file was used. 3. Can you confirm the contents of the config file found in the previous step?
o

Oliver Nguyen

06/15/2022, 10:21 AM
I have the same issue. 1. The Flytectl version that I have installed is 0.6.0
Copy code
➜  cookbook git:(master) flytectl version
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
{
  "App": "flytectl",
  "Build": "e92f7fc",
  "Version": "0.6.0",
  "BuildTime": "2022-06-15 17:22:44.272166 +0700 +07 m=+0.061320352"
}%
2. re-run the
pyflyte
command with more verbose logs
Copy code
➜  cookbook git:(master) FLYTE_SDK_LOGGING_LEVEL=20 pyflyte run --remote core/flyte_basics/hello_world.py:my_wf
{"asctime": "2022-06-15 17:18:09,655", "name": "flytekit", "levelname": "INFO", "message": "Setting protocol to file"}
{"asctime": "2022-06-15 17:18:10,462", "name": "flytekit", "levelname": "INFO", "message": "Setting protocol to file"}
{"asctime": "2022-06-15 17:18:10,463", "name": "flytekit", "levelname": "INFO", "message": "Setting protocol to file"}
{"asctime": "2022-06-15 17:18:10,463", "name": "flytekit", "levelname": "INFO", "message": "Setting protocol to file"}
{"asctime": "2022-06-15 17:18:10,465", "name": "flytekit", "levelname": "INFO", "message": "We won't register bigquery handler for structured dataset because we can't find the packages google-cloud-bigquery-storage and google-cloud-bigquery"}
E0615 17:18:10.556420000 4376362496 <http://ssl_transport_security.cc:1504]|ssl_transport_security.cc:1504]>    Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
E0615 17:18:10.558737000 4376362496 <http://ssl_transport_security.cc:1504]|ssl_transport_security.cc:1504]>    Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
{"asctime": "2022-06-15 17:18:10,559", "name": "flytekit.cli", "levelname": "INFO", "message": "Flyte Client configured -> localhost:30081 in secure mode."}
{"asctime": "2022-06-15 17:18:10,560", "name": "flytekit", "levelname": "INFO", "message": "Setting protocol to file"}
{"asctime": "2022-06-15 17:18:10,567", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1655288290.567373000\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3129,\"referenced_errors\":[{\"created\":\"@1655288290.567372000\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":163,\"grpc_status\":14}]}\"\n>, sleeping 200ms and retrying"}
{"asctime": "2022-06-15 17:18:10,771", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1655288290.770694000\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3129,\"referenced_errors\":[{\"created\":\"@1655288290.770693000\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":163,\"grpc_status\":14}]}\"\n>, sleeping 400ms and retrying"}
Traceback (most recent call last):
  File "/usr/local/bin/pyflyte", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/flytekit/clis/sdk_in_container/run.py", line 152, in run
    wf = remote.register_script(
  File "/usr/local/lib/python3.9/site-packages/flytekit/remote/remote.py", line 517, in register_script
    upload_location, md5_version = fast_register_single_script(
  File "/usr/local/lib/python3.9/site-packages/flytekit/tools/script_mode.py", line 118, in fast_register_single_script
    upload_location = create_upload_location_fn(content_md5=md5)
  File "/usr/local/lib/python3.9/site-packages/flytekit/clients/friendly.py", line 998, in get_upload_signed_url
    return super(SynchronousFlyteClient, self).create_upload_location(
  File "/usr/local/lib/python3.9/site-packages/flytekit/clients/raw.py", line 40, in handler
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/flytekit/clients/raw.py", line 834, in create_upload_location
    return self._dataproxy_stub.CreateUploadLocation(create_upload_location_request, metadata=self._metadata)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1655288291.172324000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3129,"referenced_errors":[{"created":"@1655288291.172323000","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}"
>
k

Ketan (kumare3)

06/15/2022, 11:09 AM
@Oliver Nguyen - @Evan Sadler got it running. He missed a step after flytectl demo start - Okay it works now. Needed to add these…somehow I missed them!
Copy code
KUBECONFIG=$KUBECONFIG:/Users/evan.sadler/.kube/config:/Users/evan.sadler/.flyte/k3s/k3s.yaml
export FLYTECTL_CONFIG=/Users/evan.sadler/.flyte/config-sandbox.yaml
👍 1
e

Evan Sadler

06/15/2022, 1:30 PM
Thanks for the help!
n

Niels Bantilan

06/15/2022, 1:44 PM
I think it some point we deleted the lines that explicitly point to those export statements:
see the
Note
admonition
e

Evan Sadler

06/15/2022, 1:50 PM
Cool. Are you sure they print out every time?
n

Niels Bantilan

06/15/2022, 1:50 PM
after running
flytectl demo start
I’m not seeing the export statements in my stdout… did we remove them, or did we just forget to add it?? @Ketan (kumare3) @Prafulla Mahindrakar @Yuvraj
Copy code
👨‍💻 Flyte is ready! Flyte UI is available at <http://localhost:30080/console> 🚀 🚀 🎉
However with
flytectl sandbox start
I see the following:
Copy code
👨‍💻 Flyte is ready! Flyte UI is available at <http://localhost:30081/console> 🚀 🚀 🎉
Add KUBECONFIG and FLYTECTL_CONFIG to your environment variable
export KUBECONFIG=$KUBECONFIG:/Users/nielsbantilan/.kube/config:/Users/nielsbantilan/.flyte/k3s/k3s.yaml
export FLYTECTL_CONFIG=/Users/nielsbantilan/.flyte/config-sandbox.yaml
We should add those export statements to
flytectl demo start
e

Evan Sadler

06/15/2022, 1:51 PM
This is exactly what happened…I tried the sandbox start
in a desperate attempt to get it working haha
n

Niels Bantilan

06/15/2022, 1:52 PM
cool, yeah so
sandbox
and
demo
are effectively the same for the most part, but
demo
starts up way faster.
e

Evan Sadler

06/15/2022, 1:57 PM
Sweet. @Niels Bantilan I am struggling to deploy my workflow to the local flyte sanbox. Would you be able to help me quickly debug? I can get the hello world demo working, but when I run my own workflow it fails with this:
Copy code
File "/Users/evan.sadler/miniconda3/envs/hbo/lib/python3.9/site-packages/flytekit/clients/raw.py", line 55, in handler
    raise _user_exceptions.FlyteEntityNotExistException(e)
flytekit.exceptions.user.FlyteEntityNotExistException: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.NOT_FOUND
	details = "missing entity of type WORKFLOW with identifier project:"flytesnacks" domain:"development" name:"flyte.workflows.retention_example.run_all_model_configs" version:"6ik5rx9XN7kPjwiZU3xdxg==" "
	debug_error_string = "{"created":"@1655301388.336135000","description":"Error received from peer ipv6:[::1]:30081","file":"src/core/lib/surface/call.cc","file_line":904,"grpc_message":"missing entity of type WORKFLOW with identifier project:"flytesnacks" domain:"development" name:"flyte.workflows.retention_example.run_all_model_configs" version:"6ik5rx9XN7kPjwiZU3xdxg==" ","grpc_status":5}"
n

Niels Bantilan

06/15/2022, 1:58 PM
sure, I need to step out for an hour, will be back to help debug then!
e

Evan Sadler

06/15/2022, 1:58 PM
Thanks!
p

Prafulla Mahindrakar

06/15/2022, 2:24 PM
@Niels Bantilan it seems we didn’t have those exports. will add those . @Evan Sadler seems the registration step wasnt done. Can you try registering your workflow using
flytectl register  files
meanwhile https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_register_files.html
👍 1
e

Evan Sadler

06/15/2022, 2:54 PM
@Prafulla Mahindrakar I did this and I am getting the same error. THe dynamic workflows look like they are being registered as tasks in the UI. Any ideas?
Copy code
$ pyflyte --pkgs flyte.workflows serialize --local-source-root ./ --image retention:64c7ae739661a211efcfc0f977e56a4c782893a7 workflows -f _pb_output/
$flytectl register files _pb_output/* -p flytesnacks -d development --version v1
Copy code
--------------------------------------------------------------------------- --------- ------------------------------
| NAME (6)                                                                  | STATUS  | ADDITIONAL INFO              |
 --------------------------------------------------------------------------- --------- ------------------------------
| _pb_output/0_components.example.snowflake_unload_1.pb                     | Success | Successfully registered file |
 --------------------------------------------------------------------------- --------- ------------------------------
| _pb_output/1_components.example.preprocess_1.pb                           | Success | Successfully registered file |
 --------------------------------------------------------------------------- --------- ------------------------------
| _pb_output/2_components.example.lightgbm_train_1.pb                       | Success | Successfully registered file |
 --------------------------------------------------------------------------- --------- ------------------------------
| _pb_output/3_components.example.classifier_validation_1.pb                | Success | Successfully registered file |
 --------------------------------------------------------------------------- --------- ------------------------------
| _pb_output/4_flyte.workflows.retention_example.run_config_1.pb            | Success | Successfully registered file |
 --------------------------------------------------------------------------- --------- ------------------------------
| _pb_output/5_flyte.workflows.retention_example.run_all_model_configs_1.pb | Success | Successfully registered file |
 --------------------------------------------------------------------------- --------- ------------------------------
Maybe I need to wrap the @dyanmic in @workflow
k

Ketan (kumare3)

06/15/2022, 2:57 PM
@dynamic is a task
Once you execute it will spawn a workflow Right
They are at the boundary of a task And a workflow
Tasks materialize inputs and so dynamic are tasks
But dynamic when encounters another task it delays the execution and constructs a Graph first and hence is a workflow
n

Niels Bantilan

06/15/2022, 3:02 PM
yep, we call them “dynamic workflows” because the end state is a workflow, but it uses tasks under the hood to dynamically generate the workflow at runtime.
e

Evan Sadler

06/15/2022, 4:24 PM
Sweet! Thanks okay I see the workflow
I am so close. I need to point to a specific docker image that I built locally using
./docker_build_and_tag.sh
. I want all my components to use it by default. Do I need to use the
--image
tag to use it?
Copy code
pyflyte run --remote --image retention:64c7ae739661a211efcfc0f977e56a4c782893a7 flyte/workflows/retention_example.py run_all_model_configs_wf
When I use
--image
I get this
Copy code
[1/1] currentAttempt done. Last Error: USER::containers with unready status: [f5891b9ba9721454887a-n0-0]|Back-off pulling image "retention:64c7ae739661a211efcfc0f977e56a4c782893a7"
When I don’t, packages are not installed.
I am currently trying to setup a local registry, which might help
Copy code
docker run -d -p 5000:5000 --restart=always --name local-registry registry:2
e

Eduardo Apolinario (eapolinario)

06/15/2022, 4:50 PM
@Evan Sadler, both the demo and the sandbox clusters run inside a docker container, so you have to build your image using the docker daemon that's running inside this outer docker container. You can exec to the right image via
flytectl
and build your image there (in order to exec into the right container do
flytectl sandbox exec -- <command>
)
and then you'll be able to use the
--image
in the call to
pyflyte run
as you are doing
e

Evan Sadler

06/15/2022, 4:52 PM
Thank you! I am confused on how to use
flytectl sandbox exec --
. Would you mind providing more information?
e

Eduardo Apolinario (eapolinario)

06/15/2022, 4:59 PM
sure, maybe an image will help: https://docs.flyte.org/en/latest/deployment/sandbox.html#flyte-sandbox-as-a-single-docker-container. So in your case, you should be able to just do
flytectl sandbox exec -- ./docker_build_and_tag.sh
, assuming that the script is present in the dir you used to start your sandbox.
the same applies to the demo cluster (i.e. if you started your cluster using
flytectl demo start
, then you will be able to do
flytectl demo exec -- <command>
)
e

Evan Sadler

06/15/2022, 5:03 PM
Oh great. I will restart the the sandbox and try this!
Thank you @Eduardo Apolinario (eapolinario) and folks for all the help. I am ready for tomorrow. Hopefully this goes well and we can setup more formal introductions with the HBO team.
❤️ 2
t

Tim Bauer

07/11/2022, 11:54 AM
I also wanted to finally try out the new
pyflyte run --remote
and get this Error:
Copy code
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNIMPLEMENTED
        details = "Received http2 header with status: 404"
        debug_error_string = "{"created":"@1657540045.111245779","description":"Error received from peer ipv4:10.194.0.6:443","file":"src/core/lib/surface/call.cc","file_line":952,"grpc_message":"Received http2 header with status: 404","grpc_status":12}"
We're running flyteadmin 1.1.25 currently - it should already implement the
CreateUploadLocationRequest
where this fails
k

Ketan (kumare3)

07/11/2022, 2:39 PM
Ingress?
p

Prafulla Mahindrakar

07/11/2022, 2:40 PM
Hi @Tim Bauer can you check your ingress config for DataProxy specifically these https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/templates/common/ingress.yaml#L27-L50
👀 1
t

Tim Bauer

07/12/2022, 7:40 AM
Ah yes we have an older manifest missing those paths, thank you both! 🙏
👍 1
I noticed this was not updated in https://github.com/flyteorg/flyte/blob/master/kustomize/overlays/eks/flyte/ingress/ingress_grpc.yaml which currently use as reference.
p

Prafulla Mahindrakar

07/12/2022, 3:45 PM
Its been hard maintaining both kustomize and helm and seems that might be the reason you are seeing those differences. Helm is the recommended way we are asking users to manage there flyte deployments.
k

Ketan (kumare3)

07/13/2022, 1:00 AM
Cc @jeev at some point freenome had volunteered to maintain kustomize
👍 1
j

jeev

07/13/2022, 1:18 AM
we’re happy to at least while we’re using kustomize internally
🙏 2
329 Views