Chris Green
07/31/2023, 5:41 PMFabio Grätz
07/31/2023, 6:21 PMChris Green
07/31/2023, 6:29 PMFabio Grätz
08/15/2023, 5:06 PMMohd Shahid Khan Afridi
09/11/2023, 12:11 PMflytekit
and identity_aware_proxy
in a virtual environment.
with ~/.flyte/config.yaml as
admin:
authType: pkce
insecure: false
endpoint: dns:///<url>.com
proxyCommand: [ "sh","-c","curl --silent --data client_id=<hash>.<http://apps.googleusercontent.com|apps.googleusercontent.com> --data client_secret=<hash> --data refresh_token=<token> --data grant_type=refresh_token <https://oauth2.googleapis.com/token> | gcloud auth print-identity-token" ]
Now when I am trying to access flyteRemote I still see below error
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAUTHENTICATED
details = "Invalid IAP credentials: empty token"
debug_error_string = "UNKNOWN:Error received from peer ipv4:34.98.102.88:443 {created_time:"2023-09-07T19:15:56.354327+05:30", grpc_status:16,
grpc_message:"Invalid IAP credentials: empty token"}"
>
Fabio Grätz
09/11/2023, 12:13 PM"Invalid IAP credentials: empty token"
this means that the request doesn’t even make it through the IAP in the first place.Mohd Shahid Khan Afridi
09/11/2023, 12:13 PMFabio Grätz
09/11/2023, 12:14 PMMohd Shahid Khan Afridi
09/11/2023, 12:14 PMFabio Grätz
09/11/2023, 12:14 PMMohd Shahid Khan Afridi
09/11/2023, 12:15 PMfrom flytekit.remote import FlyteRemote
from flytekit.configuration import Config
# FlyteRemote object is the main entrypoint to API
remote = FlyteRemote(config=Config.auto())
# Fetch launch plan
flyte_lp = remote.fetch_launch_plan(
project="moj-ml-workflows-flyte-example",
domain="development",
name="hello_world_ui_initiatable_lp",
version="f2534dc8747d2f604736ab1a93f6891c4bee7bab",
)
print(flyte_lp)
Fabio Grätz
09/11/2023, 1:09 PMMohd Shahid Khan Afridi
09/11/2023, 1:30 PMCompletedProcess(args=['sh', '-c', 'curl --silent --data client_id=... --data client_secret=... --data refresh_token=... --data grant_type=refresh_token <https://oauth2.googleapis.com/token> | gcloud auth print-identity-token'], returncode=0, stdout='eyJhbGciOiJ...\n', stderr='')
Fabio Grätz
09/11/2023, 1:45 PMdecode_jwt() {
jq -R 'split(".") | select(length > 0) | .[0],.[1] | @base64d | fromjson' <<< "$1"
}
I added this function to my zshrc to decode the tokens.Mohd Shahid Khan Afridi
09/11/2023, 1:48 PMFabio Grätz
09/11/2023, 2:40 PMMohd Shahid Khan Afridi
09/11/2023, 2:46 PM> Can you pls check:
1. That the audience is the webapp client id
2. That the email is set in the token1. By audience client id you mean the one used on the flyte backend? thats already webapp. The one thats provided as client_id is desktop client 2. The decoded token does contain the email address, and its the same email I am able to authenticate while accessing the webapp using browser
proxyCommand
you are using in your config file ?Fabio Grätz
09/12/2023, 7:31 AMadmin:
endpoint: dns:///my-flyte-domain.com
insecure: false
insecureSkipVerify: true
authType: Pkce
proxyCommand: ["flyte-iap", "generate-user-id-token", "--desktop_client_id", "<fill>", "--desktop_client_secret_gcp_secret_name", "<fill>", "--webapp_client_id", "<"fill web app client id used by IAP>, "--project", "fill"]
logger:
show-source: true
level: 0
storage:
type: stow
stow:
kind: google
config:
json: ""
project_id: <fill project id>
scopes: <https://www.googleapis.com/auth/devstorage.read_write>
both the azp and aud has same value which is client_idThe audience definitely should be the webapp client id used by IAP, not the desktop client id.
Mohd Shahid Khan Afridi
09/12/2023, 7:44 AMflyte-iap
, with updated proxy command it failing
ERROR:root:Failed to generate token from command ['flyte-iap', 'generate-user-id-token', '--desktop_client_id', '978053777....<http://apps.googleusercontent.com|apps.googleusercontent.com>', '--desktop_client_secret_gcp_secret_name', 'GOCSPX-ES...UMtExBw', '--webapp_client_id', '978053777608-noce0....<http://apps.googleusercontent.com|apps.googleusercontent.com>', '--project', 'prj-...']
Fabio Grätz
09/12/2023, 7:44 AMMohd Shahid Khan Afridi
09/12/2023, 7:55 AMFabio Grätz
09/12/2023, 7:56 AMMohd Shahid Khan Afridi
09/12/2023, 1:35 PMflyte-iap generate-service-account-id-token --webapp_client_id $AUDIENCE --service_account_key /Users/mohd.afridi/Downloads/sa.json
2. Once token generated by above command I am able to hit
curl --verbose --header 'Authorization: Bearer $TOKEN' https://$DNS/api/v1/projects
╭────────────────────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────────────────────╮
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:274 in continuation │
│ │
│ ❱ 274 │ │ │ │ response, call = self._thunk(new_method).with_call( │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:301 in with_call │
│ │
│ ❱ 301 │ │ return self._with_call(request, │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:287 in _with_call │
│ │
│ ❱ 287 │ │ call = self._interceptor.intercept_unary_unary(continuation, │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/flytekit/clients/grpc_utils/auth_interceptor.py:64 in intercept_unary_unary │
│ │
│ ❱ 64 │ │ │ if e.code() == grpc.StatusCode.UNAUTHENTICATED or e.code() == grpc.StatusCod │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'AuthenticationError' object has no attribute 'code'
Fabio Grätz
09/12/2023, 1:42 PMMohd Shahid Khan Afridi
09/12/2023, 2:05 PME0912 19:33:14.174233000 140704355395328 <http://hpack_parser.cc:853]|hpack_parser.cc:853]> Error parsing metadata: error=invalid value key=content-type value=text/plain; charset=utf-8
╭────────────────────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────────────────────╮
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:274 in continuation │
│ │
│ ❱ 274 │ │ │ │ response, call = self._thunk(new_method).with_call( │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:301 in with_call │
│ │
│ ❱ 301 │ │ return self._with_call(request, │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:290 in _with_call │
│ │
│ ❱ 290 │ │ return call.result(), call │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_channel.py:379 in result │
│ │
│ ❱ 379 │ │ raise self │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:274 in continuation │
│ │
│ ❱ 274 │ │ │ │ response, call = self._thunk(new_method).with_call( │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:301 in with_call │
│ │
│ ❱ 301 │ │ return self._with_call(request, │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:290 in _with_call │
│ │
│ ❱ 290 │ │ return call.result(), call │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_channel.py:379 in result │
│ │
│ ❱ 379 │ │ raise self │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:274 in continuation │
│ │
│ ❱ 274 │ │ │ │ response, call = self._thunk(new_method).with_call( │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_channel.py:1043 in with_call │
│ │
│ ❱ 1043 │ │ return _end_unary_response_blocking(state, call, True, None) │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_channel.py:910 in _end_unary_response_blocking │
│ │
│ ❱ 910 │ │ raise _InactiveRpcError(state) # pytype: disable=not-instantiable │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNIMPLEMENTED
details = "Received http2 header with status: 404"
debug_error_string = "UNKNOWN:Error received from peer ipv4:34.98.102.88:443 {grpc_message:"Received http2 header with status: 404", grpc_status:12,
created_time:"2023-09-12T19:33:15.813448+05:30"}"
Fabio Grätz
09/12/2023, 2:06 PMMohd Shahid Khan Afridi
09/12/2023, 2:08 PMFabio Grätz
09/12/2023, 2:08 PMMohd Shahid Khan Afridi
09/12/2023, 2:11 PMFabio Grätz
09/12/2023, 4:49 PMMohd Shahid Khan Afridi
09/12/2023, 4:53 PM│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:274 in continuation │
│ │
│ ❱ 274 │ │ │ │ response, call = self._thunk(new_method).with_call( │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:301 in with_call │
│ │
│ ❱ 301 │ │ return self._with_call(request, │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:287 in _with_call │
│ │
│ ❱ 287 │ │ call = self._interceptor.intercept_unary_unary(continuation, │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/flytekit/clients/grpc_utils/auth_interceptor.py:65 in intercept_unary_unary │
│ │
│ ❱ 65 │ │ │ │ self._authenticator.refresh_credentials() │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/flytekit/clients/auth/authenticator.py:141 in refresh_credentials │
│ │
│ ❱ 141 │ │ self._initialize_auth_client() │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/flytekit/clients/auth/authenticator.py:117 in _initialize_auth_client │
│ │
│ ❱ 117 │ │ │ cfg = self._cfg_store.get_client_config() │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/flytekit/clients/auth_helper.py:39 in get_client_config │
│ │
│ ❱ 39 │ │ public_client_config = metadata_service.GetPublicClientConfig(PublicClientAuthCo │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:247 in __call__ │
│ │
│ ❱ 247 │ │ response, ignored_call = self._with_call(request, │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:290 in _with_call │
│ │
│ ❱ 290 │ │ return call.result(), call │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_channel.py:379 in result │
│ │
│ ❱ 379 │ │ raise self │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_interceptor.py:274 in continuation │
│ │
│ ❱ 274 │ │ │ │ response, call = self._thunk(new_method).with_call( │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_channel.py:1043 in with_call │
│ │
│ ❱ 1043 │ │ return _end_unary_response_blocking(state, call, True, None) │
│ │
│ /Users/mohd.afridi/tutorials/flyte-wf/IAP/venv/lib/python3.8/site-packages/grpc/_channel.py:910 in _end_unary_response_blocking │
│ │
│ ❱ 910 │ │ raise _InactiveRpcError(state) # pytype: disable=not-instantiable │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNIMPLEMENTED
details = "unknown service flyteidl.service.AuthMetadataService"
debug_error_string = "UNKNOWN:Error received from peer ipv4:34.98.102.88:443 {grpc_message:"unknown service flyteidl.service.AuthMetadataService",
grpc_status:12, created_time:"2023-09-21T13:23:14.795652+05:30"}"
Fabio Grätz
09/21/2023, 8:34 AMMohd Shahid Khan Afridi
09/21/2023, 8:34 AMName: flyte-virtualservice
Namespace: flyte
Labels: <none>
Annotations: <none>
API Version: <http://networking.istio.io/v1beta1|networking.istio.io/v1beta1>
Kind: VirtualService
Metadata:
Creation Timestamp: 2023-09-21T06:25:01Z
Generation: 2
Managed Fields:
API Version: <http://networking.istio.io/v1beta1|networking.istio.io/v1beta1>
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:<http://kubectl.kubernetes.io/last-applied-configuration|kubectl.kubernetes.io/last-applied-configuration>:
f:spec:
.:
f:gateways:
f:hosts:
f:http:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2023-09-21T08:00:48Z
Resource Version: 287301262
UID: eb6ea899-f990-4f23-88f4-8e0687fc1d5e
Spec:
Gateways:
istio-system/default-gateway
Hosts:
<http://flyte-moj-ml-workflows.sharechat.com|flyte-moj-ml-workflows.sharechat.com>
Http:
Match:
Uri:
Prefix: /console
Name: console-routes
Route:
Destination:
Host: flyteconsole
Port:
Number: 80
Match:
Uri:
Prefix: /api
Uri:
Prefix: /healthcheck
Uri:
Prefix: /v1/*
Uri:
Prefix: /.well-known
Uri:
Prefix: /login
Uri:
Prefix: /logout
Uri:
Prefix: /callback
Uri:
Prefix: /me
Uri:
Prefix: /config
Uri:
Prefix: /oauth2
Name: admin-routes
Route:
Destination:
Host: flyteadmin
Port:
Number: 80
Match:
Uri:
Prefix: /flyteidl.service.SignalService
Uri:
Prefix: /flyteidl.service.AdminService
Uri:
Prefix: /flyteidl.service.DataProxyService
Uri:
Prefix: /flyteidl.service.AuthMetadataService
Uri:
Prefix: /flyteidl.service.AuthMetadataService/*
Uri:
Prefix: /flyteidl.service.IdentityService
Uri:
Prefix: /grpc.health.v1.Health
Name: admin-grpc-routes
Route:
Destination:
Host: flyteadmin
Port:
Number: 81
flyteidl.service.AuthMetadataService/*
just to tryFabio Grätz
09/21/2023, 8:36 AMspec:
rules:
- http:
paths:
- backend:
service:
name: istio-ingress
port:
number: 443
path: /
pathType: Prefix
This is how my ingress looks like.Mohd Shahid Khan Afridi
09/21/2023, 8:38 AMFabio Grätz
09/21/2023, 8:39 AMflyteidl.service.AuthMetadataService
is not forwarded from the load balancer to istio
• Or not from istio to flyteadminMohd Shahid Khan Afridi
09/21/2023, 8:40 AMFabio Grätz
09/21/2023, 9:41 AMresource.type=“http_load_balancer”This is the query I use
Mohd Shahid Khan Afridi
09/21/2023, 9:45 AM/
only in our GKE ingress and test grpc againFabio Grätz
09/21/2023, 11:49 AMMohd Shahid Khan Afridi
10/05/2023, 12:02 PMVersion: 1.10.0b0
beta release, firstly congratulations for that🎉.
Did you see any issue with released version? I have installed it on my local setup, I see the behaviour for the setup I did with the changes from your PR and this release is different. Flyte Remote started working on PR changes being installed locally but same thing with the beta release is failing_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNKNOWN
details = "Stream removed"
debug_error_string = "UNKNOWN:Error received from peer {created_time:"2023-10-05T17:15:29.44161+05:30", grpc_status:2,
grpc_message:"Stream removed"}"
Fabio Grätz
10/06/2023, 12:10 PMpip show flytekit
actually shows the 1.10 beta release?Mohd Shahid Khan Afridi
10/06/2023, 12:11 PM(venv) mohd.afridi@MTPL-5642 moj-ml-workflows % pip show flytekit
Name: flytekit
Version: 1.10.0b0
Summary: Flyte SDK for Python
Home-page: <https://github.com/flyteorg/flytekit>
Author: None
Fabio Grätz
10/06/2023, 12:11 PM"proxy-authorization"
header is included in the request and IAP denies it.Mohd Shahid Khan Afridi
10/06/2023, 12:13 PMflyte-iap generate-service-account-id-token --webapp_client_id .....
Fabio Grätz
10/06/2023, 12:18 PMMohd Shahid Khan Afridi
10/06/2023, 12:19 PMpip install flytekitplugins-identity-aware-proxy==v1.10.0b0
pip install flytekit==v1.10.0b0
pip install flytekit==v1.10.0b0
pip install flytekitplugins-identity-aware-proxy==v1.10.0b0
Fabio Grätz
10/06/2023, 1:24 PMpip show flytekit
shows the correct version. Not sure what Python wizardry this is 🤔Mohd Shahid Khan Afridi
10/06/2023, 3:07 PMFabio Grätz
10/06/2023, 9:44 PMgo.mod
of flytectl write a replace statement for flyteidl to the local path of your flyteidl checkout at the bottom of the go mod file (replace <http://github.com/flyteorg/flyteidl|github.com/flyteorg/flyteidl> => ../<path to flyteidl>
), do go mod tidy
in the flytectl repo and then make compile
.Mohd Shahid Khan Afridi
10/25/2023, 9:30 AMv1.10.0
, its failing for me. Even the reordering the install flow isn't working this time
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNKNOWN
details = "Stream removed"
debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"Stream removed", grpc_status:2,
created_time:"2023-10-25T14:59:04.485449+05:30"}"
>
Fabio Grätz
10/26/2023, 4:20 PMMohd Shahid Khan Afridi
10/30/2023, 8:44 AM