https://flyte.org
Join Slack
TLDR; 403 forbidden errors using *flytectl* (oidc and oauth2 enabled with Okta) Hi everyone! I’m st...
# flyte-support
e
TLDR; 403 forbidden errors using flytectl (oidc and oauth2 enabled with Okta) Hi everyone! I’m struggling to configure 
flytectl
I deployed Flyte with Helm on GKE cluster using OIDC and OAuth2 with Okta. The ingress is 
flyte.my.domain
exposed by GKE ingress controlle. The console works perfectly. This is how the 
config.yaml
looks like:
Copy code
admin:
  endpoint: dns:///flyte.my.domain
  authType: Pkce
  insecure: false
I keep getting this error: 
PermissionDenied desc = unexpected HTTP status code received from server: 403 (Forbidden); malformed header: missing HTTP content-type
Am I missing something? Would appreciate your help! Thanks
t
can you post a redacted version of your auth config please?
like the config map as it looks from admin.
e
@thankful-minister-83577 sure! I followed the documentation for Flyte Authentication, so we have in Okta 3 applications: Flyte (Flyteadmin), Flytectl and Flytepropeller. The auth config (from 
flyte-admin-base-config
configmap):
Copy code
server.yaml: |
    auth:
      appAuth:
        authServerType: External
        externalAuthServer:
          allowedAudience: <https://flyte.my.domain>
          baseUrl: <https://xxx.okta.com/oauth2/xxx>
          metadataUrl: .well-known/oauth-authorization-server
        thirdPartyConfig:
          flyteClient:
            clientId: xxx
            redirectUri: <http://localhost:53593/callback>
            scopes:
            - offline
            - all
      authorizedUris:
      - <https://flyte.my.domain>
      - <http://flyteadmin:80>
      - <http://flyteadmin.flyte.svc.cluster.local:80>
      userAuth:
        openId:
          baseUrl: <https://xxx.okta.com/oauth2/xxx>
          clientId: xxx
          scopes:
          - profile
          - openid
          - offline_access
By the way, I added these following specs 
allowedAudience
and 
metadataUrl
to my auth configuration although it wasn’t specified in Flyte documentation. Otherwise I got JWT authentication errors in the admin and the scheduler was failing (crashloopback status)…
t
okta shouldn’t need audience, we only added that for auth 0
metadataurl should be needed yeah
can you add this to your flytectl config?
logger: show-source: true level: 5
and try again
e
yes sure
Copy code
➜  ~ flytectl get project
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2023-05-30T18:17:39+03:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2023-05-30T18:17:39+03:00"}
{"json":{"src":"viper.go:400"},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2023-05-30T18:17:39+03:00"}
{"json":{"src":"client.go:63"},"level":"info","msg":"Initialized Admin client","ts":"2023-05-30T18:17:39+03:00"}
{"json":{"src":"auth_interceptor.go:67"},"level":"debug","msg":"Request failed due to [rpc error: code = PermissionDenied desc = unexpected HTTP status code received from server: 403 (Forbidden); malformed header: missing HTTP content-type]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2023-05-30T18:17:41+03:00"}
Error: rpc error: code = PermissionDenied desc = unexpected HTTP status code received from server: 403 (Forbidden); malformed header: missing HTTP content-type
{"json":{"src":"main.go:13"},"level":"error","msg":"rpc error: code = PermissionDenied desc = unexpected HTTP status code received from server: 403 (Forbidden); malformed header: missing HTTP content-type","ts":"2023-05-30T18:17:41+03:00"}
t
any logs on the admin side?
e
flyteadmin logs:
Copy code
{"json":{"src":"handlers.go:248"},"level":"info","msg":"Failed to parse Access Token from context. Will attempt to find IDToken. Error: [JWT_VERIFICATION_FAILED] Could not retrieve bearer token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2023-05-30T15:19:39Z"}
{"json":{"src":"handlers.go:194"},"level":"debug","msg":"gRPC server info in logging interceptor [00uw6098hCJ7P4hXU5d6]method [/flyteidl.service.AdminService/ListProjects]\n","ts":"2023-05-30T15:19:40Z"}
{"json":{"src":"handlers.go:194"},"level":"debug","msg":"gRPC server info in logging interceptor [00uw6098hCJ7P4hXU5d6]method [/flyteidl.service.IdentityService/UserInfo]\n","ts":"2023-05-30T15:19:40Z"}
{"json":{"src":"handlers.go:194"},"level":"debug","msg":"gRPC server info in logging interceptor [00uw6098hCJ7P4hXU5d6]method [/flyteidl.service.AdminService/GetVersion]\n","ts":"2023-05-30T15:19:40Z"}
2023/05/30 15:19:40 /go/pkg/mod/gorm.io/gorm@v1.24.1-0.20221019064659-5dd2bb482755/callbacks.go:134
[44.060ms] [rows:5] SELECT * FROM "projects" WHERE state != 1 ORDER BY identifier asc
t
easier to see what’s happening in python side i think
can you install this pr?
Copy code
pip install --no-deps -U --force-reinstall  "<https://github.com/flyteorg/flytekit/archive/><sha>.zip#egg=flytekit"
and then run
Copy code
$ FLYTE_SDK_LOGGING_LEVEL=10 python
from flyteidl.admin.project_pb2 import ProjectListRequest
from flytekit.remote.remote import FlyteRemote
from flytekit.configuration import Config
rr = FlyteRemote(Config.auto())
rr.client.list_projects(ProjectListRequest())
e
ok
Copy code
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.PERMISSION_DENIED
        details = "Received http2 header with status: 403"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Received http2 header with status: 403", grpc_status:7,
created_time:"2023-05-30T20:53:33.194957+03:00"}"
BTW, I’m using Flyte helm chart version 1.6.0
t
that’s it?
did it print the log line that was added?
wait this is 403…
i don’t think i’ve seen 403
are you sure okta is set up correctly?
e
Yes, for the console it works with Okta
@thankful-minister-83577 Thanks for the help! The issue was solved. Just to share it with everyone, we got 403 errors since Cloudflare blocks gRPC connections. We had to enable it under Network > Enable gRPC
🚀 2
t
thanks, that is good to know
180 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by