Thread
#ask-the-community
    e

    Evan Sadler

    3 months ago
    I am trying to use the flyte demo locally (OSX intel chip) using the
    --remote
    command. I am running into an issue. I am not sure where to start looking, so any advice is much appreciated. Inside of a new venv
    $git clone <https://github.com/flyteorg/flytesnacks>
    $cd flytesnacks/cookbook
    $pip install -r core/requirements.txt
    
    $flytectl demo start
    $pyflyte run --remote core/flyte_basics/hello_world.py:my_wf
    And then I get this
    Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
    
    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    	status = StatusCode.UNAVAILABLE
    	details = "failed to connect to all addresses"
    	debug_error_string = "{"created":"@1655239314.572674000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3129,"referenced_errors":[{"created":"@1655239314.572673000","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}"
    Ketan (kumare3)

    Ketan (kumare3)

    3 months ago
    hmm weird
    there is no SSL
    Eduardo Apolinario (eapolinario)

    Eduardo Apolinario (eapolinario)

    3 months ago
    @Evan Sadler, you shouldn't be seeing SSL errors in this case. A few things to double-check:1. can you double-check which version of flytectl you're running? 2. Can you re-run the
    pyflyte
    command with more verbose logs (run
    FLYTE_SDK_LOGGING_LEVEL=20 pyflyte run --remote core/flyte_basics/hello_world.py:my_wf
    ). These logs will tell you which config file was used. 3. Can you confirm the contents of the config file found in the previous step?
    o

    Oliver Nguyen

    3 months ago
    I have the same issue.1. The Flytectl version that I have installed is 0.6.0
    ➜  cookbook git:(master) flytectl version
    INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
    {
      "App": "flytectl",
      "Build": "e92f7fc",
      "Version": "0.6.0",
      "BuildTime": "2022-06-15 17:22:44.272166 +0700 +07 m=+0.061320352"
    }%
    2. re-run the
    pyflyte
    command with more verbose logs
      cookbook git:(master) FLYTE_SDK_LOGGING_LEVEL=20 pyflyte run --remote core/flyte_basics/hello_world.py:my_wf
    {"asctime": "2022-06-15 17:18:09,655", "name": "flytekit", "levelname": "INFO", "message": "Setting protocol to file"}
    {"asctime": "2022-06-15 17:18:10,462", "name": "flytekit", "levelname": "INFO", "message": "Setting protocol to file"}
    {"asctime": "2022-06-15 17:18:10,463", "name": "flytekit", "levelname": "INFO", "message": "Setting protocol to file"}
    {"asctime": "2022-06-15 17:18:10,463", "name": "flytekit", "levelname": "INFO", "message": "Setting protocol to file"}
    {"asctime": "2022-06-15 17:18:10,465", "name": "flytekit", "levelname": "INFO", "message": "We won't register bigquery handler for structured dataset because we can't find the packages google-cloud-bigquery-storage and google-cloud-bigquery"}
    E0615 17:18:10.556420000 4376362496 <http://ssl_transport_security.cc:1504]|ssl_transport_security.cc:1504]>    Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
    E0615 17:18:10.558737000 4376362496 <http://ssl_transport_security.cc:1504]|ssl_transport_security.cc:1504]>    Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
    {"asctime": "2022-06-15 17:18:10,559", "name": "flytekit.cli", "levelname": "INFO", "message": "Flyte Client configured -> localhost:30081 in secure mode."}
    {"asctime": "2022-06-15 17:18:10,560", "name": "flytekit", "levelname": "INFO", "message": "Setting protocol to file"}
    {"asctime": "2022-06-15 17:18:10,567", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1655288290.567373000\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3129,\"referenced_errors\":[{\"created\":\"@1655288290.567372000\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":163,\"grpc_status\":14}]}\"\n>, sleeping 200ms and retrying"}
    {"asctime": "2022-06-15 17:18:10,771", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1655288290.770694000\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3129,\"referenced_errors\":[{\"created\":\"@1655288290.770693000\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":163,\"grpc_status\":14}]}\"\n>, sleeping 400ms and retrying"}
    Traceback (most recent call last):
      File "/usr/local/bin/pyflyte", line 8, in <module>
        sys.exit(main())
      File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
        return self.main(*args, **kwargs)
      File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main
        rv = self.invoke(ctx)
      File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
        return __callback(*args, **kwargs)
      File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
        return f(get_current_context(), *args, **kwargs)
      File "/usr/local/lib/python3.9/site-packages/flytekit/clis/sdk_in_container/run.py", line 152, in run
        wf = remote.register_script(
      File "/usr/local/lib/python3.9/site-packages/flytekit/remote/remote.py", line 517, in register_script
        upload_location, md5_version = fast_register_single_script(
      File "/usr/local/lib/python3.9/site-packages/flytekit/tools/script_mode.py", line 118, in fast_register_single_script
        upload_location = create_upload_location_fn(content_md5=md5)
      File "/usr/local/lib/python3.9/site-packages/flytekit/clients/friendly.py", line 998, in get_upload_signed_url
        return super(SynchronousFlyteClient, self).create_upload_location(
      File "/usr/local/lib/python3.9/site-packages/flytekit/clients/raw.py", line 40, in handler
        return fn(*args, **kwargs)
      File "/usr/local/lib/python3.9/site-packages/flytekit/clients/raw.py", line 834, in create_upload_location
        return self._dataproxy_stub.CreateUploadLocation(create_upload_location_request, metadata=self._metadata)
      File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
        return _end_unary_response_blocking(state, call, False, None)
      File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
        raise _InactiveRpcError(state)
    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    	status = StatusCode.UNAVAILABLE
    	details = "failed to connect to all addresses"
    	debug_error_string = "{"created":"@1655288291.172324000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3129,"referenced_errors":[{"created":"@1655288291.172323000","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}"
    >
    Ketan (kumare3)

    Ketan (kumare3)

    3 months ago
    @Oliver Nguyen - @Evan Sadler got it running. He missed a step after flytectl demo start - Okay it works now. Needed to add these…somehow I missed them!
    KUBECONFIG=$KUBECONFIG:/Users/evan.sadler/.kube/config:/Users/evan.sadler/.flyte/k3s/k3s.yaml
    export FLYTECTL_CONFIG=/Users/evan.sadler/.flyte/config-sandbox.yaml
    e

    Evan Sadler

    3 months ago
    Thanks for the help!
    Niels Bantilan

    Niels Bantilan

    3 months ago
    I think it some point we deleted the lines that explicitly point to those export statements:
    see the
    Note
    admonition
    e

    Evan Sadler

    3 months ago
    Cool. Are you sure they print out every time?
    Niels Bantilan

    Niels Bantilan

    3 months ago
    after running
    flytectl demo start
    I’m not seeing the export statements in my stdout… did we remove them, or did we just forget to add it?? @Ketan (kumare3) @Prafulla Mahindrakar @Yuvraj
    👨‍💻 Flyte is ready! Flyte UI is available at <http://localhost:30080/console> 🚀 🚀 🎉
    However with
    flytectl sandbox start
    I see the following:
    👨‍💻 Flyte is ready! Flyte UI is available at <http://localhost:30081/console> 🚀 🚀 🎉
    Add KUBECONFIG and FLYTECTL_CONFIG to your environment variable
    export KUBECONFIG=$KUBECONFIG:/Users/nielsbantilan/.kube/config:/Users/nielsbantilan/.flyte/k3s/k3s.yaml
    export FLYTECTL_CONFIG=/Users/nielsbantilan/.flyte/config-sandbox.yaml
    We should add those export statements to
    flytectl demo start
    e

    Evan Sadler

    3 months ago
    This is exactly what happened…I tried the sandbox start
    in a desperate attempt to get it working haha
    Niels Bantilan

    Niels Bantilan

    3 months ago
    cool, yeah so
    sandbox
    and
    demo
    are effectively the same for the most part, but
    demo
    starts up way faster.
    e

    Evan Sadler

    3 months ago
    Sweet. @Niels Bantilan I am struggling to deploy my workflow to the local flyte sanbox. Would you be able to help me quickly debug? I can get the hello world demo working, but when I run my own workflow it fails with this:
    File "/Users/evan.sadler/miniconda3/envs/hbo/lib/python3.9/site-packages/flytekit/clients/raw.py", line 55, in handler
        raise _user_exceptions.FlyteEntityNotExistException(e)
    flytekit.exceptions.user.FlyteEntityNotExistException: <_InactiveRpcError of RPC that terminated with:
    	status = StatusCode.NOT_FOUND
    	details = "missing entity of type WORKFLOW with identifier project:"flytesnacks" domain:"development" name:"flyte.workflows.retention_example.run_all_model_configs" version:"6ik5rx9XN7kPjwiZU3xdxg==" "
    	debug_error_string = "{"created":"@1655301388.336135000","description":"Error received from peer ipv6:[::1]:30081","file":"src/core/lib/surface/call.cc","file_line":904,"grpc_message":"missing entity of type WORKFLOW with identifier project:"flytesnacks" domain:"development" name:"flyte.workflows.retention_example.run_all_model_configs" version:"6ik5rx9XN7kPjwiZU3xdxg==" ","grpc_status":5}"
    Niels Bantilan

    Niels Bantilan

    3 months ago
    sure, I need to step out for an hour, will be back to help debug then!
    e

    Evan Sadler

    3 months ago
    Thanks!
    p

    Prafulla Mahindrakar

    3 months ago
    @Niels Bantilan it seems we didn’t have those exports. will add those . @Evan Sadler seems the registration step wasnt done. Can you try registering your workflow using
    flytectl register  files
    meanwhile https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_register_files.html
    e

    Evan Sadler

    3 months ago
    @Prafulla Mahindrakar I did this and I am getting the same error. THe dynamic workflows look like they are being registered as tasks in the UI. Any ideas?
    $ pyflyte --pkgs flyte.workflows serialize --local-source-root ./ --image retention:64c7ae739661a211efcfc0f977e56a4c782893a7 workflows -f _pb_output/
    $flytectl register files _pb_output/* -p flytesnacks -d development --version v1
    --------------------------------------------------------------------------- --------- ------------------------------
    | NAME (6)                                                                  | STATUS  | ADDITIONAL INFO              |
     --------------------------------------------------------------------------- --------- ------------------------------
    | _pb_output/0_components.example.snowflake_unload_1.pb                     | Success | Successfully registered file |
     --------------------------------------------------------------------------- --------- ------------------------------
    | _pb_output/1_components.example.preprocess_1.pb                           | Success | Successfully registered file |
     --------------------------------------------------------------------------- --------- ------------------------------
    | _pb_output/2_components.example.lightgbm_train_1.pb                       | Success | Successfully registered file |
     --------------------------------------------------------------------------- --------- ------------------------------
    | _pb_output/3_components.example.classifier_validation_1.pb                | Success | Successfully registered file |
     --------------------------------------------------------------------------- --------- ------------------------------
    | _pb_output/4_flyte.workflows.retention_example.run_config_1.pb            | Success | Successfully registered file |
     --------------------------------------------------------------------------- --------- ------------------------------
    | _pb_output/5_flyte.workflows.retention_example.run_all_model_configs_1.pb | Success | Successfully registered file |
     --------------------------------------------------------------------------- --------- ------------------------------
    Maybe I need to wrap the @dyanmic in @workflow
    Ketan (kumare3)

    Ketan (kumare3)

    3 months ago
    @dynamic is a task
    Once you execute it will spawn a workflow Right
    They are at the boundary of a task And a workflow
    Tasks materialize inputs and so dynamic are tasks
    But dynamic when encounters another task it delays the execution and constructs a Graph first and hence is a workflow
    Niels Bantilan

    Niels Bantilan

    3 months ago
    yep, we call them “dynamic workflows” because the end state is a workflow, but it uses tasks under the hood to dynamically generate the workflow at runtime.
    e

    Evan Sadler

    3 months ago
    Sweet! Thanks okay I see the workflow
    I am so close. I need to point to a specific docker image that I built locally using
    ./docker_build_and_tag.sh
    . I want all my components to use it by default. Do I need to use the
    --image
    tag to use it?
    pyflyte run --remote --image retention:64c7ae739661a211efcfc0f977e56a4c782893a7 flyte/workflows/retention_example.py run_all_model_configs_wf
    When I use
    --image
    I get this
    [1/1] currentAttempt done. Last Error: USER::containers with unready status: [f5891b9ba9721454887a-n0-0]|Back-off pulling image "retention:64c7ae739661a211efcfc0f977e56a4c782893a7"
    When I don’t, packages are not installed.
    I am currently trying to setup a local registry, which might help
    docker run -d -p 5000:5000 --restart=always --name local-registry registry:2
    Eduardo Apolinario (eapolinario)

    Eduardo Apolinario (eapolinario)

    3 months ago
    @Evan Sadler, both the demo and the sandbox clusters run inside a docker container, so you have to build your image using the docker daemon that's running inside this outer docker container. You can exec to the right image via
    flytectl
    and build your image there (in order to exec into the right container do
    flytectl sandbox exec -- <command>
    )
    and then you'll be able to use the
    --image
    in the call to
    pyflyte run
    as you are doing
    e

    Evan Sadler

    3 months ago
    Thank you! I am confused on how to use
    flytectl sandbox exec --
    . Would you mind providing more information?
    Eduardo Apolinario (eapolinario)

    Eduardo Apolinario (eapolinario)

    3 months ago
    sure, maybe an image will help: https://docs.flyte.org/en/latest/deployment/sandbox.html#flyte-sandbox-as-a-single-docker-container. So in your case, you should be able to just do
    flytectl sandbox exec -- ./docker_build_and_tag.sh
    , assuming that the script is present in the dir you used to start your sandbox.
    the same applies to the demo cluster (i.e. if you started your cluster using
    flytectl demo start
    , then you will be able to do
    flytectl demo exec -- <command>
    )
    e

    Evan Sadler

    3 months ago
    Oh great. I will restart the the sandbox and try this!
    Thank you @Eduardo Apolinario (eapolinario) and folks for all the help. I am ready for tomorrow. Hopefully this goes well and we can setup more formal introductions with the HBO team.
    Tim Bauer

    Tim Bauer

    2 months ago
    I also wanted to finally try out the new
    pyflyte run --remote
    and get this Error:
    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
            status = StatusCode.UNIMPLEMENTED
            details = "Received http2 header with status: 404"
            debug_error_string = "{"created":"@1657540045.111245779","description":"Error received from peer ipv4:10.194.0.6:443","file":"src/core/lib/surface/call.cc","file_line":952,"grpc_message":"Received http2 header with status: 404","grpc_status":12}"
    We're running flyteadmin 1.1.25 currently - it should already implement the
    CreateUploadLocationRequest
    where this fails
    Ketan (kumare3)

    Ketan (kumare3)

    2 months ago
    Ingress?
    p

    Prafulla Mahindrakar

    2 months ago
    Hi @Tim Bauer can you check your ingress config for DataProxy specifically these https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/templates/common/ingress.yaml#L27-L50
    Tim Bauer

    Tim Bauer

    2 months ago
    Ah yes we have an older manifest missing those paths, thank you both! 🙏
    I noticed this was not updated in https://github.com/flyteorg/flyte/blob/master/kustomize/overlays/eks/flyte/ingress/ingress_grpc.yaml which currently use as reference.
    p

    Prafulla Mahindrakar

    2 months ago
    Its been hard maintaining both kustomize and helm and seems that might be the reason you are seeing those differences. Helm is the recommended way we are asking users to manage there flyte deployments.
    Ketan (kumare3)

    Ketan (kumare3)

    2 months ago
    Cc @jeev at some point freenome had volunteered to maintain kustomize
    j

    jeev

    2 months ago
    we’re happy to at least while we’re using kustomize internally