https://flyte.org logo
#ask-the-community
Title
# ask-the-community
c

Cornelis Boon

03/22/2024, 3:03 PM
Hey I've setup my cluster with ingress that's basic auth secured. I am not sure how I should setup the config for pyflyte though so I can run/connect to it? Things I've tried are (and then running `pyflyte run remote-task ..)`: โ€ข set
admin.endpoint
to
dns:///user:pass@domain
-> could not not contact DNS servers (perhaps I need to mveo the username/pass into some header. Unclear how to do this via config file) โ€ข set
admin.endpoint
to
<http://user:pass@domain>
-> invalid ipv6 url โ€ข set
admin.endpoint
to
<dns://domain>
-> Trying to connect an http1.x server
Tried setting
admin.endpoint
to
<dns://domain:8089>
and
admin.authorizationHeader
to
Basic BASE64(user:pass)
. Now getting:
Copy code
Failed to connect to remote host: FD Shutdown
Also getting this if I don't set the authHeader, so not sure. ๐Ÿค”
When trying to connect via grpcurl, I get:
Copy code
Failed to dial target host "domain:8089": context deadline exceeded
Both with and without the auth. HTTP ingress seems to work fine btw. Can reach console page via
domain/console
I do see the domain being resolved to the correct IP, so something is going on. Just don't see anything happening in the ingress or flyte logs
Ah. Seems like this is because TLS isn't properly setup ๐Ÿค”
y

Yee

03/22/2024, 4:12 PM
itโ€™s just dns:///domain
if youโ€™re using an idp and openidconnect it should just route you to your idp
c

Cornelis Boon

03/22/2024, 4:12 PM
I'm using simple basic auth for now, hence the auth header
y

Yee

03/22/2024, 4:13 PM
the idea is your secret is inside the
FAKE_SECRET_NAME
env var
c

Cornelis Boon

03/22/2024, 4:13 PM
Ah, gotcha
y

Yee

03/22/2024, 4:13 PM
so as to not store a secret inside a config file of course
c

Cornelis Boon

03/22/2024, 4:15 PM
Nice that's convenient
I'm planning to store the whole config file in my secret manager and then mount it wherever is necessary. That should be secure enough too, right? OR should I keep the authHeader a separate secret and ideally still use the FAKE_SECRET_NAME env approach?
Followup, I managed to setup TLS correctly and am now able to hit my flyte deployment with an `endpoint: dns:///my-domain`with no auth configured. However, can't seem to figure out the correct way to configure pyflyte for if I secure my deployment with basic-auth (in nginx ingress). This is my current config. Still getting 401s when I try to run
pyflyte run remote-task list
for example. When I try to do this programatically by using
FlyteRemote
, I also get 401 when I try to fetch the workflow/task. Then I dug deeper, and it seems to be an issue with how the client sets itself up. If I do e.g.
Copy code
remote = FlyteRemote(config=Config.auto(), default_project="flytesnacks",default_domain="development")
flyte_wf = remote.fetch_workflow(name="<http://workflows.example.wf|workflows.example.wf>")
I get the same issue, but with a stacktrace (in ๐Ÿงต ). It seems like it's trying to make an unauthenticated request to an
AuthMetadataService
to get some config before actually making authenticated requests. But since that service is likely also behind the basic-auth ingress, that request never makes it through. Does this mean basic-auth as I've setup is currently not supported or am I missing something?\
Copy code
โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Traceback (most recent call last) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”‚ in <module>:1                                                                                                                                                                   โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ <root>/anaconda3/envs/flyte/lib/python3.10/site-packages/flytekit/remote/remote.py:378 in fetch_workflow                                                    โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ โฑ  378 โ”‚   โ”‚   โ”‚   self.client.list_workflows_paginated,                                                                                                                        โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ <root>/anaconda3/envs/flyte/lib/python3.10/site-packages/flytekit/remote/remote.py:236 in client                                                            โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ โฑ  236 โ”‚   โ”‚   โ”‚   self._client = SynchronousFlyteClient(self.config.platform, **self._kwargs)                                                                                  โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ <root>/anaconda3/envs/flyte/lib/python3.10/site-packages/flytekit/clients/raw.py:50 in __init__                                                             โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ โฑ  50 โ”‚   โ”‚   โ”‚   cfg, upgrade_channel_to_authenticated(cfg, upgrade_channel_to_proxy_authenti                                                                                  โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ <root>/anaconda3/envs/flyte/lib/python3.10/site-packages/flytekit/clients/auth_helper.py:140 in upgrade_channel_to_authenticated                            โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ โฑ 140 โ”‚   authenticator = get_authenticator(cfg, RemoteClientConfigStore(in_channel))                                                                                           โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ <root>/anaconda3/envs/flyte/lib/python3.10/site-packages/flytekit/clients/auth_helper.py:75 in get_authenticator                                            โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ โฑ  75 โ”‚   โ”‚   return ClientCredentialsAuthenticator(                                                                                                                            โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ <root>/anaconda3/envs/flyte/lib/python3.10/site-packages/flytekit/clients/auth/authenticator.py:213 in __init__                                             โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ โฑ 213 โ”‚   โ”‚   cfg = cfg_store.get_client_config()                                                                                                                               โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ <root>/anaconda3/envs/flyte/lib/python3.10/site-packages/flytekit/clients/auth_helper.py:38 in get_client_config                                            โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ โฑ  38 โ”‚   โ”‚   public_client_config = metadata_service.GetPublicClientConfig(PublicClientAuthCo                                                                                  โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ <root>/anaconda3/envs/flyte/lib/python3.10/site-packages/grpc/_interceptor.py:277 in __call__                                                               โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ โฑ 277 โ”‚   โ”‚   response, ignored_call = self._with_call(                                                                                                                         โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ <root>/anaconda3/envs/flyte/lib/python3.10/site-packages/grpc/_interceptor.py:332 in _with_call                                                             โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ โฑ 332 โ”‚   โ”‚   return call.result(), call                                                                                                                                        โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ <root>/anaconda3/envs/flyte/lib/python3.10/site-packages/grpc/_channel.py:439 in result                                                                     โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ โฑ  439 โ”‚   โ”‚   raise self                                                                                                                                                       โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ <root>/anaconda3/envs/flyte/lib/python3.10/site-packages/grpc/_interceptor.py:315 in continuation                                                           โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ โฑ 315 โ”‚   โ”‚   โ”‚   โ”‚   response, call = self._thunk(new_method).with_call(                                                                                                       โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ <root>/anaconda3/envs/flyte/lib/python3.10/site-packages/grpc/_channel.py:1193 in with_call                                                                 โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ โฑ 1193 โ”‚   โ”‚   return _end_unary_response_blocking(state, call, True, None)                                                                                                     โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ <root>/anaconda3/envs/flyte/lib/python3.10/site-packages/grpc/_channel.py:1005 in _end_unary_response_blocking                                              โ”‚
โ”‚                                                                                                                                                                                 โ”‚
โ”‚ โฑ 1005 โ”‚   โ”‚   raise _InactiveRpcError(state)  # pytype: disable=not-instantiable                                                                                               โ”‚
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAUTHENTICATED
        details = "Received http2 header with status: 401"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-03-24T13:07:36.398921517+01:00", grpc_status:16, grpc_message:"Received http2 header with 
status: 401"}"
Ah. Solved it by adding
/flyteidl.service.AuthMetadataService
to ingress-nginx's configmap
no-auth-locations
However, now getting:
Copy code
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNIMPLEMENTED
        details = "unknown service flyteidl.service.AuthMetadataService"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-03-24T13:38:14.959242639+01:00", grpc_status:12, grpc_message:"unknown service 
flyteidl.service.AuthMetadataService"}"
I'm guessing I need to enable auth for this to work? But if i try that I get:
Copy code
OIDC base URL required when authentication is enabled
which I obviously do not have and it won't allow me to pass an empty string either ๐Ÿค”
Which worked instantly ๐Ÿ‘