https://flyte.org logo
Powered by
#flyte-on-gcp
Title
# flyte-on-gcp
c

Chris Green

07/31/2023, 5:41 PM
I’m also trying to get flyte setup on GCP. My company has a strict requirement to use IAP everywhere. We’ve been able to get authentication to work on the console but I can’t get access via the flytectl to work. Any tips here? recent thread https://flyte-org.slack.com/archives/CP2HDHKE1/p1690818970921269
f

Fabio Grätz

07/31/2023, 6:21 PM
Hey, I’m working on an RFC to propose changes to flytectl and flytekit to make IAP work. We also plan to upstream a reference deployment with IAP on GCP. Will be 2 weeks until the rfc though, currently on a holiday
c

Chris Green

07/31/2023, 6:29 PM
Can I ask anyone else for help in the interim?
f

Fabio Grätz

08/15/2023, 5:06 PM
1 PR missing that documents how to configure the helm chart. Will work on that one next.
@Matthew Corley
m

Mohd Shahid Khan Afridi

09/11/2023, 12:11 PM
Hi @Fabio Grätz, I have pulled your changes installed 
flytekit
and 
identity_aware_proxy
in a virtual environment. with ~/.flyte/config.yaml as
Copy code
admin:
 authType: pkce
 insecure: false
 endpoint: dns:///<url>.com
 proxyCommand: [ "sh","-c","curl --silent --data client_id=<hash>.<http://apps.googleusercontent.com|apps.googleusercontent.com> --data client_secret=<hash> --data refresh_token=<token> --data grant_type=refresh_token <https://oauth2.googleapis.com/token> | gcloud auth print-identity-token" ]
Now when I am trying to access flyteRemote I still see below error
Copy code
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAUTHENTICATED
        details = "Invalid IAP credentials: empty token"
        debug_error_string = "UNKNOWN:Error received from peer ipv4:34.98.102.88:443 {created_time:"2023-09-07T19:15:56.354327+05:30", grpc_status:16, 
grpc_message:"Invalid IAP credentials: empty token"}"
>
The client_id and secret are of a desktop_app client created in gcp credentials
f

Fabio Grätz

09/11/2023, 12:13 PM
"Invalid IAP credentials: empty token"
this means that the request doesn’t even make it through the IAP in the first place.
In which order did you install flytekit and the plugin?
m

Mohd Shahid Khan Afridi

09/11/2023, 12:13 PM
Yes
f

Fabio Grätz

09/11/2023, 12:14 PM
It matters because the plugin depends on flytekit and I was in a situation where I thought I had installed flytekit from local dir but actually the plugin uninstalled it again and installed from pypi.
So the proxy-auth header logic wasn’t actually there.
m

Mohd Shahid Khan Afridi

09/11/2023, 12:14 PM
I installed flytekit first and then plugin
f

Fabio Grätz

09/11/2023, 12:14 PM
Pls try the other way round, I think i ran into exactly this before
m

Mohd Shahid Khan Afridi

09/11/2023, 12:15 PM
ooh okay let me try
Still same 😞 , is my configuration looks fine. Anything else i need for this follow to be tirggered?
code I am trying to run
Copy code
from flytekit.remote import FlyteRemote
from flytekit.configuration import Config

# FlyteRemote object is the main entrypoint to API
remote = FlyteRemote(config=Config.auto())

# Fetch launch plan
flyte_lp = remote.fetch_launch_plan(
    project="moj-ml-workflows-flyte-example", 
    domain="development",
    name="hello_world_ui_initiatable_lp",
    version="f2534dc8747d2f604736ab1a93f6891c4bee7bab",
)
print(flyte_lp)
I can see your changes in the flytekit package installed in my venv
f

Fabio Grätz

09/11/2023, 1:09 PM
Could you pls put a breakpoint in the external command authenticator here (just in my branch) to make sure the external command is acutally run and that a correct token is obtained?
String should start with “eY…”
m

Mohd Shahid Khan Afridi

09/11/2023, 1:30 PM
(Pdb) p output
Copy code
CompletedProcess(args=['sh', '-c', 'curl --silent --data client_id=... --data client_secret=... --data refresh_token=... --data grant_type=refresh_token <https://oauth2.googleapis.com/token> | gcloud auth print-identity-token'], returncode=0, stdout='eyJhbGciOiJ...\n', stderr='')
f

Fabio Grätz

09/11/2023, 1:45 PM
Copy code
decode_jwt() {
    jq -R 'split(".") | select(length > 0) | .[0],.[1] | @base64d | fromjson' <<< "$1"
}
I added this function to my zshrc to decode the tokens.
Can you pls check: 1. That the audience is the webapp client id 2. That the email is set in the token
Also, can you please check that the respective email has “IAP secured webapp user” permissions?
Also which time zone are you in? Happy to jump on a call to debug (before wednesday I have little time though :/)
m

Mohd Shahid Khan Afridi

09/11/2023, 1:48 PM
I am working in IST but no such constrain I can manage
let me know when you are available?
f

Fabio Grätz

09/11/2023, 2:40 PM
Around noon on Wednsday your time would work for me
Can you pls check the questions above? Maybe we can solve it async even beforehand 🙂
m

Mohd Shahid Khan Afridi

09/11/2023, 2:46 PM
Yeah I am looking into it
> Can you pls check:
1. That the audience is the webapp client id
2. That the email is set in the token
1. By audience client id you mean the one used on the flyte backend? thats already webapp. The one thats provided as client_id is desktop client 2. The decoded token does contain the email address, and its the same email I am able to authenticate while accessing the webapp using browser
One thing I noticed in my decoded JWT is that both the azp and aud has same value which is client_id Can you share the 
proxyCommand
you are using in your config file ?
f

Fabio Grätz

09/12/2023, 7:31 AM
Copy code
admin:
  endpoint: dns:///my-flyte-domain.com
  insecure: false
  insecureSkipVerify: true
  authType: Pkce
  proxyCommand: ["flyte-iap", "generate-user-id-token", "--desktop_client_id", "<fill>", "--desktop_client_secret_gcp_secret_name", "<fill>", "--webapp_client_id", "<"fill web app client id used by IAP>, "--project",  "fill"]
logger:
  show-source: true
  level: 0
storage:
  type: stow
  stow:
    kind: google
    config:
      json: ""
      project_id: <fill project id>
      scopes: <https://www.googleapis.com/auth/devstorage.read_write>
This is how my client config looks like
both the azp and aud has same value which is client_id
The audience definitely should be the webapp client id used by IAP, not the desktop client id.
Ok checked in my token. azp is the desktop client id, aud is the webapp client id.
m

Mohd Shahid Khan Afridi

09/12/2023, 7:44 AM
Do I need some additional step to install 
flyte-iap
, with updated proxy command it failing
Copy code
ERROR:root:Failed to generate token from command ['flyte-iap', 'generate-user-id-token', '--desktop_client_id', '978053777....<http://apps.googleusercontent.com|apps.googleusercontent.com>', '--desktop_client_secret_gcp_secret_name', 'GOCSPX-ES...UMtExBw', '--webapp_client_id', '978053777608-noce0....<http://apps.googleusercontent.com|apps.googleusercontent.com>', '--project', 'prj-...']
f

Fabio Grätz

09/12/2023, 7:44 AM
If you installed the flytekit-iap plugin the command should be there
Try running the command in your cli directly, not via the task config, to see whether it works there and how the token looks like.
Did you create the GCP secret with the desktop client secret? (See readme in the plugin)
m

Mohd Shahid Khan Afridi

09/12/2023, 7:55 AM
Aah I completely missed this guide, sorry 😞
f

Fabio Grätz

09/12/2023, 7:56 AM
No worries 🙂
m

Mohd Shahid Khan Afridi

09/12/2023, 1:35 PM
UPDATE: Some work pending on the user-id-token side to be done but able to make some progress with service account 1. Below command successfully generated token
Copy code
flyte-iap generate-service-account-id-token --webapp_client_id $AUDIENCE --service_account_key /Users/mohd.afridi/Downloads/sa.json
2. Once token generated by above command I am able to hit
Copy code
curl --verbose --header 'Authorization: Bearer $TOKEN' https://$DNS/api/v1/projects
But same command in config is still not working
Error:
Copy code
╭────────────────────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────────────────────╮
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:274 in continuation                                            │
│                                                                                                                                                                │
│ ❱ 274 │   │   │   │   response, call = self._thunk(new_method).with_call(                                                                                      │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:301 in with_call                                               │
│                                                                                                                                                                │
│ ❱ 301 │   │   return self._with_call(request,                                                                                                                  │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:287 in _with_call                                              │
│                                                                                                                                                                │
│ ❱ 287 │   │   call = self._interceptor.intercept_unary_unary(continuation,                                                                                     │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/flytekit/clients/grpc_utils/auth_interceptor.py:64 in intercept_unary_unary         │
│                                                                                                                                                                │
│ ❱ 64 │   │   │   if e.code() == grpc.StatusCode.UNAUTHENTICATED or e.code() == grpc.StatusCod                                                                  │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'AuthenticationError' object has no attribute 'code'
f

Fabio Grätz

09/12/2023, 1:42 PM
Can you pls put a breakpoint into the CommandAuthenticator to check what the result of the shell call is? 🤔
m

Mohd Shahid Khan Afridi

09/12/2023, 2:05 PM
Finally 403 error gone ...atleast progressed to a new error 🙂
Copy code
E0912 19:33:14.174233000 140704355395328 <http://hpack_parser.cc:853]|hpack_parser.cc:853]>          Error parsing metadata: error=invalid value key=content-type value=text/plain; charset=utf-8
╭────────────────────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────────────────────╮
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:274 in continuation                                            │
│                                                                                                                                                                │
│ ❱ 274 │   │   │   │   response, call = self._thunk(new_method).with_call(                                                                                      │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:301 in with_call                                               │
│                                                                                                                                                                │
│ ❱ 301 │   │   return self._with_call(request,                                                                                                                  │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:290 in _with_call                                              │
│                                                                                                                                                                │
│ ❱ 290 │   │   return call.result(), call                                                                                                                       │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_channel.py:379 in result                                                      │
│                                                                                                                                                                │
│ ❱  379 │   │   raise self                                                                                                                                      │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:274 in continuation                                            │
│                                                                                                                                                                │
│ ❱ 274 │   │   │   │   response, call = self._thunk(new_method).with_call(                                                                                      │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:301 in with_call                                               │
│                                                                                                                                                                │
│ ❱ 301 │   │   return self._with_call(request,                                                                                                                  │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:290 in _with_call                                              │
│                                                                                                                                                                │
│ ❱ 290 │   │   return call.result(), call                                                                                                                       │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_channel.py:379 in result                                                      │
│                                                                                                                                                                │
│ ❱  379 │   │   raise self                                                                                                                                      │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:274 in continuation                                            │
│                                                                                                                                                                │
│ ❱ 274 │   │   │   │   response, call = self._thunk(new_method).with_call(                                                                                      │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_channel.py:1043 in with_call                                                  │
│                                                                                                                                                                │
│ ❱ 1043 │   │   return _end_unary_response_blocking(state, call, True, None)                                                                                    │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_channel.py:910 in _end_unary_response_blocking                                │
│                                                                                                                                                                │
│ ❱  910 │   │   raise _InactiveRpcError(state)  # pytype: disable=not-instantiable                                                                              │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNIMPLEMENTED
        details = "Received http2 header with status: 404"
        debug_error_string = "UNKNOWN:Error received from peer ipv4:34.98.102.88:443 {grpc_message:"Received http2 header with status: 404", grpc_status:12, 
created_time:"2023-09-12T19:33:15.813448+05:30"}"
f

Fabio Grätz

09/12/2023, 2:06 PM
Maybe the routing of your ingress don’t expose some endpoints of admin? 🤔
You seem to be past the iap
m

Mohd Shahid Khan Afridi

09/12/2023, 2:08 PM
yes It looks I am past IAP now, need to look into my GKE ingress now I guess
f

Fabio Grätz

09/12/2023, 2:08 PM
Take a look at the deployment section in my readme
Let me know if any step doesn’t work in there. Tried to be detailed but maybe sth still isn’t clear
m

Mohd Shahid Khan Afridi

09/12/2023, 2:11 PM
Sure, for now we configured GKE ingress manually looking at all the paths from ngnix ingress helm created. Will see if your deployment steps can help
f

Fabio Grätz

09/12/2023, 4:49 PM
You can’t put flyteadmin behind GCE ingress without using TLS between load balancer and backend.
m

Mohd Shahid Khan Afridi

09/12/2023, 4:53 PM
This works for me for the REST( tested via curl as stated here), I am still waiting for some inputs from devops team to get the exact routing and will test by adding grpc route for experimentation
After following the deployment doc(slightly modified in the last part as I already had the domain name and all set) and setting up istio. Now facing below error, It looks like the request has reached the server side
Copy code
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:274 in continuation                                            │
│                                                                                                                                                                │
│ ❱ 274 │   │   │   │   response, call = self._thunk(new_method).with_call(                                                                                      │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:301 in with_call                                               │
│                                                                                                                                                                │
│ ❱ 301 │   │   return self._with_call(request,                                                                                                                  │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:287 in _with_call                                              │
│                                                                                                                                                                │
│ ❱ 287 │   │   call = self._interceptor.intercept_unary_unary(continuation,                                                                                     │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/flytekit/clients/grpc_utils/auth_interceptor.py:65 in intercept_unary_unary         │
│                                                                                                                                                                │
│ ❱ 65 │   │   │   │   self._authenticator.refresh_credentials()                                                                                                 │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/flytekit/clients/auth/authenticator.py:141 in refresh_credentials                   │
│                                                                                                                                                                │
│ ❱ 141 │   │   self._initialize_auth_client()                                                                                                                   │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/flytekit/clients/auth/authenticator.py:117 in _initialize_auth_client               │
│                                                                                                                                                                │
│ ❱ 117 │   │   │   cfg = self._cfg_store.get_client_config()                                                                                                    │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/flytekit/clients/auth_helper.py:39 in get_client_config                             │
│                                                                                                                                                                │
│ ❱  39 │   │   public_client_config = metadata_service.GetPublicClientConfig(PublicClientAuthCo                                                                 │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:247 in __call__                                                │
│                                                                                                                                                                │
│ ❱ 247 │   │   response, ignored_call = self._with_call(request,                                                                                                │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:290 in _with_call                                              │
│                                                                                                                                                                │
│ ❱ 290 │   │   return call.result(), call                                                                                                                       │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_channel.py:379 in result                                                      │
│                                                                                                                                                                │
│ ❱  379 │   │   raise self                                                                                                                                      │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:274 in continuation                                            │
│                                                                                                                                                                │
│ ❱ 274 │   │   │   │   response, call = self._thunk(new_method).with_call(                                                                                      │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_channel.py:1043 in with_call                                                  │
│                                                                                                                                                                │
│ ❱ 1043 │   │   return _end_unary_response_blocking(state, call, True, None)                                                                                    │
│                                                                                                                                                                │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_channel.py:910 in _end_unary_response_blocking                                │
│                                                                                                                                                                │
│ ❱  910 │   │   raise _InactiveRpcError(state)  # pytype: disable=not-instantiable                                                                              │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNIMPLEMENTED
        details = "unknown service flyteidl.service.AuthMetadataService"
        debug_error_string = "UNKNOWN:Error received from peer ipv4:34.98.102.88:443 {grpc_message:"unknown service flyteidl.service.AuthMetadataService", 
grpc_status:12, created_time:"2023-09-21T13:23:14.795652+05:30"}"
@Fabio Grätz ^^
f

Fabio Grätz

09/21/2023, 8:34 AM
To me it looks like the request reached the istio ingress but that the forwarding from there to flyteadmin isn’t working as it should.
Can you please paste the state of your virtual service?
m

Mohd Shahid Khan Afridi

09/21/2023, 8:34 AM
image.png
Copy code
Name:         flyte-virtualservice
Namespace:    flyte
Labels:       <none>
Annotations:  <none>
API Version:  <http://networking.istio.io/v1beta1|networking.istio.io/v1beta1>
Kind:         VirtualService
Metadata:
  Creation Timestamp:  2023-09-21T06:25:01Z
  Generation:          2
  Managed Fields:
    API Version:  <http://networking.istio.io/v1beta1|networking.istio.io/v1beta1>
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:<http://kubectl.kubernetes.io/last-applied-configuration|kubectl.kubernetes.io/last-applied-configuration>:
      f:spec:
        .:
        f:gateways:
        f:hosts:
        f:http:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2023-09-21T08:00:48Z
  Resource Version:  287301262
  UID:               eb6ea899-f990-4f23-88f4-8e0687fc1d5e
Spec:
  Gateways:
    istio-system/default-gateway
  Hosts:
    <http://flyte-moj-ml-workflows.sharechat.com|flyte-moj-ml-workflows.sharechat.com>
  Http:
    Match:
      Uri:
        Prefix:  /console
    Name:        console-routes
    Route:
      Destination:
        Host:  flyteconsole
        Port:
          Number:  80
    Match:
      Uri:
        Prefix:  /api
      Uri:
        Prefix:  /healthcheck
      Uri:
        Prefix:  /v1/*
      Uri:
        Prefix:  /.well-known
      Uri:
        Prefix:  /login
      Uri:
        Prefix:  /logout
      Uri:
        Prefix:  /callback
      Uri:
        Prefix:  /me
      Uri:
        Prefix:  /config
      Uri:
        Prefix:  /oauth2
    Name:        admin-routes
    Route:
      Destination:
        Host:  flyteadmin
        Port:
          Number:  80
    Match:
      Uri:
        Prefix:  /flyteidl.service.SignalService
      Uri:
        Prefix:  /flyteidl.service.AdminService
      Uri:
        Prefix:  /flyteidl.service.DataProxyService
      Uri:
        Prefix:  /flyteidl.service.AuthMetadataService
      Uri:
        Prefix:  /flyteidl.service.AuthMetadataService/*
      Uri:
        Prefix:  /flyteidl.service.IdentityService
      Uri:
        Prefix:  /grpc.health.v1.Health
    Name:        admin-grpc-routes
    Route:
      Destination:
        Host:  flyteadmin
        Port:
          Number:  81
I added 
flyteidl.service.AuthMetadataService/*
just to try
f

Fabio Grätz

09/21/2023, 8:36 AM
Copy code
spec:
    rules:
    - http:
        paths:
        - backend:
            service:
              name: istio-ingress
              port:
                number: 443
          path: /
          pathType: Prefix
This is how my ingress looks like.
m

Mohd Shahid Khan Afridi

09/21/2023, 8:38 AM
This backend service we created from console and loadbalancer rules are created from console in my case
Would you like to have a quick look?
f

Fabio Grätz

09/21/2023, 8:39 AM
There are two points where the traffic could fail: • 
flyteidl.service.AuthMetadataService
is not forwarded from the load balancer to istio • Or not from istio to flyteadmin
Can you check in cloud logging please? Write “http_loadbalancer”
You can see all requests there
Maybe this gives us a hint
m

Mohd Shahid Khan Afridi

09/21/2023, 8:40 AM
Let me check
cloud logging isn't capturing request logs in my case, may b i need to enable it somewhere
meanwhile snapshot of load-balancer montioring
f

Fabio Grätz

09/21/2023, 9:41 AM
resource.type=“http_load_balancer”
This is the query I use
Something I could imagine is also that only exact matches but not prefix are routed?
Mine looks like this
m

Mohd Shahid Khan Afridi

09/21/2023, 9:45 AM
Okay let me make it / only
atleast for the REST call as used in below code...its working fine. Will update path to 
/
only in our GKE ingress and test grpc again
f

Fabio Grätz

09/21/2023, 11:49 AM
If TLS is not working correctly between lb and backend, the load balancer will not deliver any grpc traffic to the istio ingress. In the lp logs, in this case, I see something like “Failed to deliver to backend”.
m

Mohd Shahid Khan Afridi

10/05/2023, 12:02 PM
Hi @Fabio Grätz, I see that the changes for IAP are out in 
Version: 1.10.0b0
beta release, firstly congratulations for that🎉. Did you see any issue with released version? I have installed it on my local setup, I see the behaviour for the setup I did with the changes from your PR and this release is different. Flyte Remote started working on PR changes being installed locally but same thing with the beta release is failing
I started getting below error
Copy code
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Stream removed"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2023-10-05T17:15:29.44161+05:30", grpc_status:2, 
grpc_message:"Stream removed"}"
While same thing is working with your PR changes being installed locally
Enabled logging on LB it seems the request is succeeding well, after first 401 all are 200 on LB side but failing on client side, while with your PR changes the single 200 after first 401 result in success on client side
@Fabio Grätz ^
f

Fabio Grätz

10/06/2023, 12:10 PM
Hey 🙂 I will test with the beta release later today
Til then, can you please sanity check that 
pip show flytekit
actually shows the 1.10 beta release?
The iap plugin simply requires flytekit as there is no released version with the new logic yet.
m

Mohd Shahid Khan Afridi

10/06/2023, 12:11 PM
Copy code
(venv) mohd.afridi@MTPL-5642 moj-ml-workflows % pip show flytekit
Name: flytekit
Version: 1.10.0b0
Summary: Flyte SDK for Python
Home-page: <https://github.com/flyteorg/flytekit>
Author: None
f

Fabio Grätz

10/06/2023, 12:11 PM
So maybe when installing the iap plugin after the flytekit beta, it installed the latest release again?
Ok 👍
“Stream removed” definitely means that no valid 
"proxy-authorization"
header is included in the request and IAP denies it.
Can you pls also test whether the proxy command in the flyte config works when executed manually in the cli?
m

Mohd Shahid Khan Afridi

10/06/2023, 12:13 PM
I am using service account one, its working file when run in terminal to generate token
Copy code
flyte-iap generate-service-account-id-token --webapp_client_id .....
same config is used in an environment where flytekit is installed using the PR, its working fine as well
f

Fabio Grätz

10/06/2023, 12:18 PM
Ok I will try to later test the beta 👍
Can’t tell what could be the reason but I need to investigate this
Which commit are you using from my PR?
m

Mohd Shahid Khan Afridi

10/06/2023, 12:19 PM
flytekit % git log -n 1 commit a713c56f10d36d99d900c671dc094acb84cf80a8 (HEAD -> fg91/feat/proxy-authentication, origin/fg91/feat/proxy-authentication) Author: Fabio Grätz <fabiogratz@googlemail.com> Date: Sun Sep 10 110331 2023 +0200 Document usage of generate-service-account-id-token subcommand
Found the issue, the order still matters 🥲.. below is working
Copy code
pip install flytekitplugins-identity-aware-proxy==v1.10.0b0
pip install flytekit==v1.10.0b0
previously I was setting up like below and it wasn't working though it was showing correct version
Copy code
pip install flytekit==v1.10.0b0
pip install flytekitplugins-identity-aware-proxy==v1.10.0b0
f

Fabio Grätz

10/06/2023, 1:24 PM
I can confirm that the order matters and that it doesn’t work if first installing the plugin then flytekit even though 
pip show flytekit
shows the correct version. Not sure what Python wizardry this is 🤔
That reminds me we need to pin flytekit>=1.10 here which will be the first release to contain the changes to auth helpers.
Wonder whether there is a chicken-egg problem though where CI will fail before this version actually exists 🤔
No chicken egg problem.
m

Mohd Shahid Khan Afridi

10/06/2023, 3:07 PM
Thanks for the changes
Is there any guide to pull your changes for flytectl using proxycommand-> https://github.com/flyteorg/flyteidl/pull/437 in my local setup ?
f

Fabio Grätz

10/06/2023, 9:44 PM
Checkout the flytectl repo, checkout the flyteidl repo, in the 
go.mod
of flytectl write a replace statement for flyteidl to the local path of your flyteidl checkout at the bottom of the go mod file (
replace  <http://github.com/flyteorg/flyteidl|github.com/flyteorg/flyteidl> => ../<path to flyteidl>
), do 
go mod tidy
in the flytectl repo and then 
make compile
.
m

Mohd Shahid Khan Afridi

10/25/2023, 9:30 AM
@Fabio Grätz did you get a chance to try your IAP changes published in 
v1.10.0
, its failing for me. Even the reordering the install flow isn't working this time
Copy code
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Stream removed"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Stream removed", grpc_status:2, 
created_time:"2023-10-25T14:59:04.485449+05:30"}"
>
f

Fabio Grätz

10/26/2023, 4:20 PM
Hey @Mohd Shahid Khan Afridi, got back from a holiday today, I will test this tomorrow and get back to you 🙂
m

Mohd Shahid Khan Afridi

10/30/2023, 8:44 AM
@Fabio Grätz , were you able to give it a try ?
5 Views
Powered by