https://flyte.org logo
#ask-the-community
Title
# ask-the-community
a

Ariel Kaspit

05/29/2023, 8:07 AM
TLDR; 403 forbidden errors using flytectl (oidc and oauth2 enabled with Okta) Hi everyone! I’m struggling to configure
flytectl
I deployed Flyte with Helm on GKE cluster using OIDC and OAuth2 with Okta. The ingress is
flyte.my.domain
exposed by GKE ingress controlle. The console works perfectly. This is how the
config.yaml
looks like:
Copy code
admin:
  endpoint: dns:///flyte.my.domain
  authType: Pkce
  insecure: false
I keep getting this error:
PermissionDenied desc = unexpected HTTP status code received from server: 403 (Forbidden); malformed header: missing HTTP content-type
Am I missing something? Would appreciate your help! Thanks
y

Yee

05/29/2023, 6:56 PM
can you post a redacted version of your auth config please?
like the config map as it looks from admin.
a

Ariel Kaspit

05/30/2023, 7:18 AM
@Yee sure! I followed the documentation for Flyte Authentication, so we have in Okta 3 applications: Flyte (Flyteadmin), Flytectl and Flytepropeller. The auth config (from
flyte-admin-base-config
configmap):
Copy code
server.yaml: |
    auth:
      appAuth:
        authServerType: External
        externalAuthServer:
          allowedAudience: <https://flyte.my.domain>
          baseUrl: <https://xxx.okta.com/oauth2/xxx>
          metadataUrl: .well-known/oauth-authorization-server
        thirdPartyConfig:
          flyteClient:
            clientId: xxx
            redirectUri: <http://localhost:53593/callback>
            scopes:
            - offline
            - all
      authorizedUris:
      - <https://flyte.my.domain>
      - <http://flyteadmin:80>
      - <http://flyteadmin.flyte.svc.cluster.local:80>
      userAuth:
        openId:
          baseUrl: <https://xxx.okta.com/oauth2/xxx>
          clientId: xxx
          scopes:
          - profile
          - openid
          - offline_access
By the way, I added these following specs
allowedAudience
and
metadataUrl
to my auth configuration although it wasn’t specified in Flyte documentation. Otherwise I got JWT authentication errors in the admin and the scheduler was failing (crashloopback status)…
y

Yee

05/30/2023, 3:13 PM
okta shouldn’t need audience, we only added that for auth 0
metadataurl should be needed yeah
can you add this to your flytectl config?
logger: show-source: true level: 5
and try again
a

Ariel Kaspit

05/30/2023, 3:17 PM
yes sure
Copy code
➜  ~ flytectl get project
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2023-05-30T18:17:39+03:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2023-05-30T18:17:39+03:00"}
{"json":{"src":"viper.go:400"},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2023-05-30T18:17:39+03:00"}
{"json":{"src":"client.go:63"},"level":"info","msg":"Initialized Admin client","ts":"2023-05-30T18:17:39+03:00"}
{"json":{"src":"auth_interceptor.go:67"},"level":"debug","msg":"Request failed due to [rpc error: code = PermissionDenied desc = unexpected HTTP status code received from server: 403 (Forbidden); malformed header: missing HTTP content-type]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2023-05-30T18:17:41+03:00"}
Error: rpc error: code = PermissionDenied desc = unexpected HTTP status code received from server: 403 (Forbidden); malformed header: missing HTTP content-type
{"json":{"src":"main.go:13"},"level":"error","msg":"rpc error: code = PermissionDenied desc = unexpected HTTP status code received from server: 403 (Forbidden); malformed header: missing HTTP content-type","ts":"2023-05-30T18:17:41+03:00"}
y

Yee

05/30/2023, 3:18 PM
any logs on the admin side?
a

Ariel Kaspit

05/30/2023, 3:21 PM
flyteadmin logs:
Copy code
{"json":{"src":"handlers.go:248"},"level":"info","msg":"Failed to parse Access Token from context. Will attempt to find IDToken. Error: [JWT_VERIFICATION_FAILED] Could not retrieve bearer token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2023-05-30T15:19:39Z"}
{"json":{"src":"handlers.go:194"},"level":"debug","msg":"gRPC server info in logging interceptor [00uw6098hCJ7P4hXU5d6]method [/flyteidl.service.AdminService/ListProjects]\n","ts":"2023-05-30T15:19:40Z"}
{"json":{"src":"handlers.go:194"},"level":"debug","msg":"gRPC server info in logging interceptor [00uw6098hCJ7P4hXU5d6]method [/flyteidl.service.IdentityService/UserInfo]\n","ts":"2023-05-30T15:19:40Z"}
{"json":{"src":"handlers.go:194"},"level":"debug","msg":"gRPC server info in logging interceptor [00uw6098hCJ7P4hXU5d6]method [/flyteidl.service.AdminService/GetVersion]\n","ts":"2023-05-30T15:19:40Z"}
2023/05/30 15:19:40 /go/pkg/mod/gorm.io/gorm@v1.24.1-0.20221019064659-5dd2bb482755/callbacks.go:134
[44.060ms] [rows:5] SELECT * FROM "projects" WHERE state != 1 ORDER BY identifier asc
y

Yee

05/30/2023, 3:50 PM
easier to see what’s happening in python side i think
can you install this pr?
Copy code
pip install --no-deps -U --force-reinstall  "<https://github.com/flyteorg/flytekit/archive/><sha>.zip#egg=flytekit"
and then run
Copy code
$ FLYTE_SDK_LOGGING_LEVEL=10 python
from flyteidl.admin.project_pb2 import ProjectListRequest
from flytekit.remote.remote import FlyteRemote
from flytekit.configuration import Config
rr = FlyteRemote(Config.auto())
rr.client.list_projects(ProjectListRequest())
a

Ariel Kaspit

05/30/2023, 5:53 PM
ok
Copy code
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.PERMISSION_DENIED
        details = "Received http2 header with status: 403"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Received http2 header with status: 403", grpc_status:7,
created_time:"2023-05-30T20:53:33.194957+03:00"}"
BTW, I’m using Flyte helm chart version 1.6.0
y

Yee

05/30/2023, 8:07 PM
that’s it?
did it print the log line that was added?
wait this is 403…
i don’t think i’ve seen 403
are you sure okta is set up correctly?
a

Ariel Kaspit

05/30/2023, 8:11 PM
Yes, for the console it works with Okta
@Yee Thanks for the help! The issue was solved. Just to share it with everyone, we got 403 errors since Cloudflare blocks gRPC connections. We had to enable it under Network > Enable gRPC
y

Yee

06/01/2023, 4:29 PM
thanks, that is good to know
4 Views