Hi! I hope everyone is well and safe I have a flyte cluster running on kubernetes. I am trying to run a simple workflow, from my machine, on that cluster. I am struggling a bit how to do that. Is there any documentation explaining how to do that?

hi @Vinicius Esperança have you had a chance to go over the getting started docs? these should walk you through how to run an execute remotely as well as set up a demo cluster locally

I didn't find how to configure flytecli to point to my cluster configuration

flytecli now reads from the same config as flytectl which should have been written in your init call above

@Vinicius Esperança you said you are runing flyte on a k8s cluster

yeah, I did use the docs to setup, but still couldnt make it work now I am able to see the console with projects but still having issues to run workflow for example, running

pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2

, using the

example.py

, I am getting

viniciusesperanca@Viniciuss-Mini flyte-deployment-aws % pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2                                 
{"asctime": "2022-06-13 09:43:43,653", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1655124223.653297000\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3218,\"referenced_errors\":[{\"created\":\"@1655124223.653297000\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":165,\"grpc_status\":14}]}\"\n>, sleeping 200ms and retrying"}

I think I did config it properly

flytectl config init --host=<hostname-with-ssl>

@Vinicius Esperança I read your previous comment. You are only port forwarding 8080. Grpc is running on a separate port

I am not forwarding. I am using my cluster configured behind my domain (flyte.sidetrek.com) as host on

flytectl config init --host=<http://flyte.sidetrek.com|flyte.sidetrek.com>

@Vinicius Esperança, can you bump the log verbosity in your call to

pyflyte run

to ensure it's using the right config file? Just set this env var in your call, e.g.:

FLYTE_SDK_LOGGING_LEVEL=20 pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2

it seems to be using correct file

{
  "asctime": "2022-06-13 15:12:22,563",
  "name": "flytekit",
  "levelname": "INFO",
  "message": "Using flytectl/YAML config /Users/viniciusesperanca/.flyte/config.yaml"
}

viniciusesperanca@Viniciuss-Mini flyte-deployment-aws % cat /Users/viniciusesperanca/.flyte/config.yaml                                          
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///flyte.sidetrek.com
  authType: Pkce
  insecure: false
logger:
  show-source: true
  level: 0%

ok, and just to be safe, can you list projects using flytectl?

flytectl get projects --config ~/.flyte/config.yaml

got this

Error: rpc error: code = Unavailable desc = upstream connect error or disconnect/reset before headers. reset reason: connection termination

@Vinicius Esperança, it looks like you're running on aws, if so, how did you configure flyte in your cluster? Did you follow the deployment guides?

indeed, using aws and yes, I’ve followed the steps the only different thing is, I am using emissary ingress (https://github.com/emissary-ingress/emissary) instead of aws one

@Yuvraj, @Prafulla Mahindrakar, can you help double check that ingress is working in this case? It's weird that

flytectl

can't reach admin.

I am also getting the same issue . We haven’t tried emissary in our internal deployments, but can we try checking the ingress logs . Also are you able to make flytectl get project call if you port forward admin and use that port directly in config.yaml

What is your emissary ingress config ? Can you share the steps that you used for installing emissary ?

Actually it look like a ingress mis configuration for grpc. Your Rest call are working fine in flyteconsole. I don’t have much knowledge of emissary.

after adding some new annotations to

Mapping

I can use

get projects

viniciusesperanca@web flyte-deployment-aws %  flytectl get projects --config ~/.flyte/config.yaml          
 --------------- --------------- --------------------------- 
| ID            | NAME          | DESCRIPTION               |
 --------------- --------------- --------------------------- 
| flyteexamples | flyteexamples | flyteexamples description |
 --------------- --------------- --------------------------- 
| flytetester   | flytetester   | flytetester description   |
 --------------- --------------- --------------------------- 
| flytesnacks   | flytesnacks   | flytesnacks description   |
 --------------- --------------- --------------------------- 
3 rows

what is the issue with workflow ?

failed to connect to all addresses

viniciusesperanca@web flyte-deployment-aws % pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2
{"asctime": "2022-06-14 09:08:51,001", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1655208531.001191000\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3218,\"referenced_errors\":[{\"created\":\"@1655208531.001189000\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":165,\"grpc_status\":14}]}\"\n>, sleeping 200ms and retrying"}
{"asctime": "2022-06-14 09:08:51,207", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1655208531.206841000\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3218,\"referenced_errors\":[{\"created\":\"@1655208531.206838000\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":165,\"grpc_status\":14}]}\"\n>, sleeping 400ms and retrying"}
Traceback (most recent call last):
  File "/opt/homebrew/bin/pyflyte", line 8, in <module>
    sys.exit(main())
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/homebrew/lib/python3.9/site-packages/flytekit/clis/sdk_in_container/run.py", line 473, in _run
    wf = remote.register_script(
  File "/opt/homebrew/lib/python3.9/site-packages/flytekit/remote/remote.py", line 536, in register_script
    upload_location, md5_bytes = fast_register_single_script(
  File "/opt/homebrew/lib/python3.9/site-packages/flytekit/tools/script_mode.py", line 117, in fast_register_single_script
    upload_location = create_upload_location_fn(content_md5=md5)
  File "/opt/homebrew/lib/python3.9/site-packages/flytekit/clients/friendly.py", line 998, in get_upload_signed_url
    return super(SynchronousFlyteClient, self).create_upload_location(
  File "/opt/homebrew/lib/python3.9/site-packages/flytekit/clients/raw.py", line 40, in handler
    return fn(*args, **kwargs)
  File "/opt/homebrew/lib/python3.9/site-packages/flytekit/clients/raw.py", line 834, in create_upload_location
    return self._dataproxy_stub.CreateUploadLocation(create_upload_location_request, metadata=self._metadata)
  File "/opt/homebrew/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/opt/homebrew/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1655208531.612898000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3218,"referenced_errors":[{"created":"@1655208531.612891000","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":165,"grpc_status":14}]}"
>

Do you have flytectl config in place ?

I think so

viniciusesperanca@web flyte-deployment-aws % cat ~/.flyte/config.yaml 
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///flyte.sidetrek.com
  authType: Pkce
  insecure: false
logger:
  show-source: true
  level: 0%

I tested pyflyte run with our cloud deployment and it worked. Not sure what’s the issue look like nginx issue but let’s wait for @Eduardo Apolinario (eapolinario)

that worked @Yuvraj thank you so much

amazing, @Vinicius Esperança. Just so the community can benefit from this investigation, you ended up using the sandbox cluster for your tests, i.e. you didn't have to configure a different ingress for your case, right?

I used the

larger apps deploy

and kept my configuration with

emissary ingress

I still don’t know which ingress mapping I am missing in order to run with

pyflyte

, but uploading workflow directly to my flyte cluster worked I will probably keep investigating this further and can post here if I find something