https://flyte.org logo
#flytekit
Title
# flytekit
v

Vinicius Esperança

06/10/2022, 10:00 PM
Hi! I hope everyone is well and safe I have a flyte cluster running on kubernetes. I am trying to run a simple workflow, from my machine, on that cluster. I am struggling a bit how to do that. Is there any documentation explaining how to do that?
I have configured to use my
port-forward
Copy code
flytectl config init --host=localhost:8080
--- fort forward is the following:
Copy code
kubectl port-forward svc/flyteadmin -n flyte 8080:80
--- when I try to run I receive the following:
Copy code
viniciusesperanca@Viniciuss-Mini flyte-deployment-aws % pyflyte run example.py wf --n 500 --mean 42 --sigma 2
E0610 19:07:30.345097000 4337059200 <http://ssl_transport_security.cc:1495]|ssl_transport_security.cc:1495]>    Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
E0610 19:07:30.812457000 4337059200 <http://ssl_transport_security.cc:1495]|ssl_transport_security.cc:1495]>    Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
DefaultNamedTupleOutput(o0=41.92207532626968, o1=2.125158116611769)
I am not really sure how to test it
k

katrina

06/10/2022, 10:11 PM
hi @Vinicius Esperança have you had a chance to go over the getting started docs? these should walk you through how to run an execute remotely as well as set up a demo cluster locally
v

Vinicius Esperança

06/10/2022, 10:18 PM
I didn't find how to configure flytecli to point to my cluster configuration
k

katrina

06/10/2022, 10:21 PM
flytecli now reads from the same config as flytectl which should have been written in your init call above
is the flyteadmin grpc port configured to 80?
k

Ketan (kumare3)

06/10/2022, 10:54 PM
@Vinicius Esperança you said you are runing flyte on a k8s cluster
you cannot simply port forward
as ssl wil not get forwarded correctly
you probably want to use ingress
please use the docs to set it up
for experimentation i recommend using the getting started to test locally
v

Vinicius Esperança

06/13/2022, 12:49 PM
yeah, I did use the docs to setup, but still couldnt make it work now I am able to see the console with projects but still having issues to run workflow for example, running
pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2
, using the
example.py
, I am getting
Copy code
viniciusesperanca@Viniciuss-Mini flyte-deployment-aws % pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2                                 
{"asctime": "2022-06-13 09:43:43,653", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1655124223.653297000\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3218,\"referenced_errors\":[{\"created\":\"@1655124223.653297000\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":165,\"grpc_status\":14}]}\"\n>, sleeping 200ms and retrying"}
I think I did config it properly
Copy code
flytectl config init --host=<hostname-with-ssl>
i am not really sure what I am missing here
k

Ketan (kumare3)

06/13/2022, 3:39 PM
@Vinicius Esperança I read your previous comment. You are only port forwarding 8080. Grpc is running on a separate port
Can you port forward that as well
v

Vinicius Esperança

06/13/2022, 4:00 PM
I am not forwarding. I am using my cluster configured behind my domain (flyte.sidetrek.com) as host on
Copy code
flytectl config init --host=<http://flyte.sidetrek.com|flyte.sidetrek.com>
e

Eduardo Apolinario (eapolinario)

06/13/2022, 5:23 PM
@Vinicius Esperança, can you bump the log verbosity in your call to
pyflyte run
to ensure it's using the right config file? Just set this env var in your call, e.g.:
Copy code
FLYTE_SDK_LOGGING_LEVEL=20 pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2
v

Vinicius Esperança

06/13/2022, 6:15 PM
it seems to be using correct file
Copy code
{
  "asctime": "2022-06-13 15:12:22,563",
  "name": "flytekit",
  "levelname": "INFO",
  "message": "Using flytectl/YAML config /Users/viniciusesperanca/.flyte/config.yaml"
}
Copy code
viniciusesperanca@Viniciuss-Mini flyte-deployment-aws % cat /Users/viniciusesperanca/.flyte/config.yaml                                          
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///flyte.sidetrek.com
  authType: Pkce
  insecure: false
logger:
  show-source: true
  level: 0%
e

Eduardo Apolinario (eapolinario)

06/13/2022, 6:22 PM
ok, and just to be safe, can you list projects using flytectl?
Copy code
flytectl get projects --config ~/.flyte/config.yaml
v

Vinicius Esperança

06/13/2022, 6:27 PM
got this
Copy code
Error: rpc error: code = Unavailable desc = upstream connect error or disconnect/reset before headers. reset reason: connection termination
e

Eduardo Apolinario (eapolinario)

06/13/2022, 10:01 PM
@Vinicius Esperança, it looks like you're running on aws, if so, how did you configure flyte in your cluster? Did you follow the deployment guides?
v

Vinicius Esperança

06/13/2022, 10:18 PM
indeed, using aws and yes, I’ve followed the steps the only different thing is, I am using emissary ingress (https://github.com/emissary-ingress/emissary) instead of aws one
but I am not sure if that is the problem though
e

Eduardo Apolinario (eapolinario)

06/14/2022, 4:35 AM
@Yuvraj, @Prafulla Mahindrakar, can you help double check that ingress is working in this case? It's weird that
flytectl
can't reach admin.
p

Prafulla Mahindrakar

06/14/2022, 4:48 AM
I am also getting the same issue . We haven’t tried emissary in our internal deployments, but can we try checking the ingress logs . Also are you able to make flytectl get project call if you port forward admin and use that port directly in config.yaml
Also can you try to use an insecure channel first and see if that communication works through ingress before enabling secure channel .
y

Yuvraj

06/14/2022, 5:05 AM
What is your emissary ingress config ? Can you share the steps that you used for installing emissary ?
v

Vinicius Esperança

06/14/2022, 10:57 AM
also I can reach
flyte console
and show default projects
I am using Pulumi btw. Maybe I am missing some of the
Mappings
? They represent same as ingress rules.
@Yuvraj tried with insecure, same error
Copy code
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1655204980.718613000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3218,"referenced_errors":[{"created":"@1655204980.718609000","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":165,"grpc_status":14}]}"
I think I am missing some inner configuration, but I am not really sure where
y

Yuvraj

06/14/2022, 11:14 AM
Actually it look like a ingress mis configuration for grpc. Your Rest call are working fine in flyteconsole. I don’t have much knowledge of emissary.
v

Vinicius Esperança

06/14/2022, 12:11 PM
after adding some new annotations to
Mapping
I can use
get projects
Copy code
viniciusesperanca@web flyte-deployment-aws %  flytectl get projects --config ~/.flyte/config.yaml          
 --------------- --------------- --------------------------- 
| ID            | NAME          | DESCRIPTION               |
 --------------- --------------- --------------------------- 
| flyteexamples | flyteexamples | flyteexamples description |
 --------------- --------------- --------------------------- 
| flytetester   | flytetester   | flytetester description   |
 --------------- --------------- --------------------------- 
| flytesnacks   | flytesnacks   | flytesnacks description   |
 --------------- --------------- --------------------------- 
3 rows
🎉 1
but still having same problem trying to run workflow
y

Yuvraj

06/14/2022, 12:11 PM
what is the issue with workflow ?
v

Vinicius Esperança

06/14/2022, 12:12 PM
failed to connect to all addresses
Copy code
viniciusesperanca@web flyte-deployment-aws % pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2
{"asctime": "2022-06-14 09:08:51,001", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1655208531.001191000\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3218,\"referenced_errors\":[{\"created\":\"@1655208531.001189000\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":165,\"grpc_status\":14}]}\"\n>, sleeping 200ms and retrying"}
{"asctime": "2022-06-14 09:08:51,207", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1655208531.206841000\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3218,\"referenced_errors\":[{\"created\":\"@1655208531.206838000\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":165,\"grpc_status\":14}]}\"\n>, sleeping 400ms and retrying"}
Traceback (most recent call last):
  File "/opt/homebrew/bin/pyflyte", line 8, in <module>
    sys.exit(main())
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/homebrew/lib/python3.9/site-packages/flytekit/clis/sdk_in_container/run.py", line 473, in _run
    wf = remote.register_script(
  File "/opt/homebrew/lib/python3.9/site-packages/flytekit/remote/remote.py", line 536, in register_script
    upload_location, md5_bytes = fast_register_single_script(
  File "/opt/homebrew/lib/python3.9/site-packages/flytekit/tools/script_mode.py", line 117, in fast_register_single_script
    upload_location = create_upload_location_fn(content_md5=md5)
  File "/opt/homebrew/lib/python3.9/site-packages/flytekit/clients/friendly.py", line 998, in get_upload_signed_url
    return super(SynchronousFlyteClient, self).create_upload_location(
  File "/opt/homebrew/lib/python3.9/site-packages/flytekit/clients/raw.py", line 40, in handler
    return fn(*args, **kwargs)
  File "/opt/homebrew/lib/python3.9/site-packages/flytekit/clients/raw.py", line 834, in create_upload_location
    return self._dataproxy_stub.CreateUploadLocation(create_upload_location_request, metadata=self._metadata)
  File "/opt/homebrew/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/opt/homebrew/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1655208531.612898000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3218,"referenced_errors":[{"created":"@1655208531.612891000","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":165,"grpc_status":14}]}"
>
y

Yuvraj

06/14/2022, 12:12 PM
Do you have flytectl config in place ?
v

Vinicius Esperança

06/14/2022, 12:14 PM
I think so
Copy code
viniciusesperanca@web flyte-deployment-aws % cat ~/.flyte/config.yaml 
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///flyte.sidetrek.com
  authType: Pkce
  insecure: false
logger:
  show-source: true
  level: 0%
y

Yuvraj

06/14/2022, 1:14 PM
I tested pyflyte run with our cloud deployment and it worked. Not sure what’s the issue look like nginx issue but let’s wait for @Eduardo Apolinario (eapolinario)
For a time being you can use our large app deployment guide https://docs.flyte.org/projects/cookbook/en/latest/auto/larger_apps/larger_apps_deploy.html , It will fix your issue
v

Vinicius Esperança

06/15/2022, 12:34 AM
that worked @Yuvraj thank you so much
e

Eduardo Apolinario (eapolinario)

06/15/2022, 1:33 AM
amazing, @Vinicius Esperança. Just so the community can benefit from this investigation, you ended up using the sandbox cluster for your tests, i.e. you didn't have to configure a different ingress for your case, right?
v

Vinicius Esperança

06/15/2022, 11:32 AM
I used the
larger apps deploy
and kept my configuration with
emissary ingress
I still don’t know which ingress mapping I am missing in order to run with
pyflyte
, but uploading workflow directly to my flyte cluster worked I will probably keep investigating this further and can post here if I find something
👍 1
151 Views