https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
c

Chris Green

07/31/2023, 3:56 PM
Hi when running 
flytectl get projects
I’m seeing the below error:
Copy code
Error: rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type "text/plain; charset=utf-8"
I can’t tell if the 404 is from a request that the ctl made or if it’s happening in a backend service. I tried to increase the log levels on backend but couldn’t find anything. Any help would be greatly appreciated! cc @Mohd Shahid Khan Afridi
y

Yee

07/31/2023, 4:30 PM
add logging level to your flytectl config?
c

Chris Green

07/31/2023, 4:30 PM
i’ve bumped it up to 1000
y

Yee

07/31/2023, 4:30 PM
Copy code
logger:
  show-source: true
  level: 5
not 1000
c

Chris Green

07/31/2023, 4:35 PM
didn’t make a difference in the admin logs
y

Yee

07/31/2023, 4:35 PM
hmm
it’s supposed to make a difference in flytectl logs
can you paste in your config file?
redact if necessary
c

Chris Green

07/31/2023, 4:36 PM
which config?
y

Yee

07/31/2023, 4:36 PM
flytectl
c

Chris Green

07/31/2023, 4:41 PM
Copy code
admin:
    insecure: false
    endpoint: dns:///<HOSTNAME>
    insecureSkipVerify: true
    authType: ExternalCommand
    command: [echo, <BEARER TOKEN>]
    authorizationHeader: "Authorization"
logger:
    show-source: true
    level: 5
We are using Google IAP to secure backend services
So I’m trying to figure out a workaround using what’s available in flyte (i.e. the external command hack)
y

Yee

07/31/2023, 4:42 PM
i think some others have gotten google iap to work btw
c

Chris Green

07/31/2023, 4:42 PM
would love to hear how!
y

Yee

07/31/2023, 4:42 PM
is there any other output from flytectl
can you paste all of it
c

Chris Green

07/31/2023, 4:44 PM
actually taking a closer look it seems i have an issue with the IAP credentials
Copy code
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2023-07-31T10:43:26-06:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2023-07-31T10:43:26-06:00"}
{"json":{"src":"viper.go:400"},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2023-07-31T10:43:26-06:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [files] updated. No update handler registered.","ts":"2023-07-31T10:43:26-06:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [console] updated. No update handler registered.","ts":"2023-07-31T10:43:26-06:00"}
{"json":{"src":"client.go:145"},"level":"warning","msg":"using insecureSkipVerify. Server's certificate chain and host name wont be verified. Caution : shouldn't be used for production usecases","ts":"2023-07-31T10:43:26-06:00"}
{"json":{"src":"client.go:63"},"level":"info","msg":"Initialized Admin client","ts":"2023-07-31T10:43:26-06:00"}
{"json":{"src":"auth_interceptor.go:86"},"level":"debug","msg":"Request failed due to [rpc error: code = Unauthenticated desc = Invalid IAP credentials: empty token]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2023-07-31T10:43:26-06:00"}
{"json":{"src":"auth_interceptor.go:91"},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2023-07-31T10:43:26-06:00"}
{"json":{"src":"client.go:145"},"level":"warning","msg":"using insecureSkipVerify. Server's certificate chain and host name wont be verified. Caution : shouldn't be used for production usecases","ts":"2023-07-31T10:43:26-06:00"}
Error: rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type "text/plain; charset=utf-8"
{"json":{"src":"main.go:13"},"level":"error","msg":"rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type \"text/plain; charset=utf-8\"","ts":"2023-07-31T10:43:27-06:00"}
Copy code
Unauthenticated desc = Invalid IAP credentials: empty token
y

Yee

07/31/2023, 4:48 PM
can you add grpc debugging?
Copy code
GRPC_TRACE=all GRPC_VERBOSITY=DEBUG
near the end it should say what it’s trying to hit. maybe one of the auth metadata endpoints
dump to a log and search for 
flyteidl
c

Chris Green

07/31/2023, 4:52 PM
where do i set these?
y

Yee

07/31/2023, 4:52 PM
just on the command line
or export as env vars
c

Chris Green

07/31/2023, 4:53 PM
didn’t make a difference in logging output
y

Yee

07/31/2023, 4:58 PM
apparently the env vars don’t work in flytectl…
can i get you to try these two endpoints in a browser
see what they return
c

Chris Green

07/31/2023, 4:59 PM
what should be the base url?
i’m going to assume it’s the endpoint in the config
in which case i get
Copy code
{
  "error": "unknown service flyteidl.service.AuthMetadataService",
  "code": 12,
  "message": "unknown service flyteidl.service.AuthMetadataService"
}
actually i updated the config to 
useAuth: true
and redeployed admin
Copy code
{
  "client_id": "flytectl",
  "redirect_uri": "<http://localhost:53593/callback>",
  "scopes": [
    "offline",
    "all"
  ],
  "authorization_metadata_key": "flyte-authorization"
}
and
Copy code
{
  "issuer": "<https://accounts.google.com>",
  "authorization_endpoint": "<https://accounts.google.com/o/oauth2/v2/auth>",
  "token_endpoint": "<https://oauth2.googleapis.com/token>",
  "response_types_supported": [
    "code",
    "token",
    "id_token",
    "code token",
    "code id_token",
    "token id_token",
    "code token id_token",
    "none"
  ],
  "scopes_supported": [
    "openid",
    "email",
    "profile"
  ],
  "token_endpoint_auth_methods_supported": [
    "client_secret_post",
    "client_secret_basic"
  ],
  "jwks_uri": "<https://www.googleapis.com/oauth2/v3/certs>",
  "code_challenge_methods_supported": [
    "plain",
    "S256"
  ],
  "grant_types_supported": [
    "authorization_code",
    "refresh_token",
    "urn:ietf:params:oauth:grant-type:device_code",
    "urn:ietf:params:oauth:grant-type:jwt-bearer"
  ],
  "device_authorization_endpoint": "<https://oauth2.googleapis.com/device/code>"
}
y

Yee

07/31/2023, 5:14 PM
try again?
c

Chris Green

07/31/2023, 5:15 PM
flytectl?
y

Yee

07/31/2023, 5:15 PM
yeah
c

Chris Green

07/31/2023, 5:16 PM
Copy code
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2023-07-31T11:15:59-06:00"}
{"json":{"src":"viper.go:400"},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2023-07-31T11:15:59-06:00"}
{"json":{"src":"client.go:145"},"level":"warning","msg":"using insecureSkipVerify. Server's certificate chain and host name wont be verified. Caution : shouldn't be used for production usecases","ts":"2023-07-31T11:15:59-06:00"}
{"json":{"src":"client.go:63"},"level":"info","msg":"Initialized Admin client","ts":"2023-07-31T11:15:59-06:00"}
{"json":{"src":"auth_interceptor.go:67"},"level":"debug","msg":"Request failed due to [rpc error: code = Unauthenticated desc = Invalid IAP credentials: empty token]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2023-07-31T11:15:59-06:00"}
{"json":{"src":"auth_interceptor.go:72"},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2023-07-31T11:15:59-06:00"}
{"json":{"src":"client.go:145"},"level":"warning","msg":"using insecureSkipVerify. Server's certificate chain and host name wont be verified. Caution : shouldn't be used for production usecases","ts":"2023-07-31T11:15:59-06:00"}
Error: authentication error! Original Error: rpc error: code = Unauthenticated desc = Invalid IAP credentials: empty token, Auth Error: pipe: too many open files
{"json":{"src":"main.go:13"},"level":"error","msg":"authentication error! Original Error: rpc error: code = Unauthenticated desc = Invalid IAP credentials: empty token, Auth Error: pipe: too many open files","ts":"2023-07-31T11:15:59-06:00"}
y

Yee

07/31/2023, 5:17 PM
nice.
maybe open a new thread in flyte-on-gcp?
i have no experience here
c

Chris Green

07/31/2023, 5:22 PM
ok will do
f

Fabio Grätz

08/15/2023, 5:17 PM
See progress here.
9 Views
Powered by