Hi Team, We have recently deployed flyte-core helm chart on our Kubernetes cluster. We are using Azure AD for user Authentication and internal auth server for app authentication. We are able to deploy all components successfully except for

flytescheduler

component. This deployment is failing at init container step with follow error:

{"json":{},"level":"warning","msg":"failed to get token: %!w(*url.Error=&{Post <http://flyteadmin:81/oauth2/token> 0xc0002b8280})","ts":"2024-07-02T12:12:04Z"}
Error: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: Post "<http://flyteadmin:81/oauth2/token>": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x06\x04\x00\x00\x00\x00\x00\x00\x05\x00\x00@\x00"

Following is the Kubernetes configmap configuration for scheduler deployment

apiVersion: v1
data:
  admin.yaml: |
    admin:
      clientId: 'flytepropeller'
      clientSecretLocation: /etc/secrets/client_secret
      endpoint: flyteadmin:81
      insecure: true
    event:
      capacity: 1000
      rate: 500
      type: admin
  db.yaml: |
    database:
      dbname: postgres
      host: postgresql
      passwordPath: /etc/db/pass.txt
      port: 5432
      username: postgres
  server.yaml: |
    scheduler:
      metricsScope: 'flyte:'
      profilerPort: 10254
kind: ConfigMap
metadata:
  name: flyte-scheduler-config

Can someone please help us on this issue ?

could you try with

admin.insecure: false

?

hi @average-finland-92144, I see the following error on using this option.

panic: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"

what about the flyte admin configmap? That init container starts an authenticated session to flyteadmin

Hi @average-finland-92144, here is the admin configmap configuration

apiVersion: v1
data:
  cluster_resources.yaml: |
    cluster_resources:
      customData:
      - production:
        - projectQuotaCpu:
            value: "5"
        - projectQuotaMemory:
            value: 4000Mi
        - defaultIamRole:
            value: <AWS_ROLE_SUBSTITUTED_HERE_>
      - staging:
        - projectQuotaCpu:
            value: "2"
        - projectQuotaMemory:
            value: 3000Mi
        - defaultIamRole:
            value: <AWS_ROLE_SUBSTITUTED_HERE_>
      - development:
        - projectQuotaCpu:
            value: "4"
        - projectQuotaMemory:
            value: 3000Mi
        - defaultIamRole:
            value: <AWS_ROLE_SUBSTITUTED_HERE_>
      refreshInterval: 5m
      standaloneDeployment: false
      templatePath: /etc/flyte/clusterresource/templates
  db.yaml: |
    database:
      dbname: flyteadmin
      host: '<AWS_POSTGRES_HOST_SUBSTITUTED_HERE_>'
      passwordPath: /etc/db/pass.txt
      port: 5432
      username: 'dbadmin'
  domain.yaml: |
    domains:
    - id: development
      name: development
    - id: staging
      name: staging
    - id: production
      name: production
  remoteData.yaml: |
    remoteData:
      region: us-east-1
      scheme: local
      signedUrls:
        durationMinutes: 3
  server.yaml: |
    auth:
      appAuth:
        selfAuthServer:
          staticClients:
            flyte-cli:
              grant_types:
              - refresh_token
              - authorization_code
              id: flyte-cli
              public: true
              redirect_uris:
              - <http://localhost:53593/callback>
              - <http://localhost:12345/callback>
              response_types:
              - code
              - token
              scopes:
              - all
              - offline
              - access_token
            flytectl:
              grant_types:
              - refresh_token
              - authorization_code
              id: flytectl
              public: true
              redirect_uris:
              - <http://localhost:53593/callback>
              - <http://localhost:12345/callback>
              response_types:
              - code
              - token
              scopes:
              - all
              - offline
              - access_token
            flytepropeller:
              client_secret: ''
              grant_types:
              - refresh_token
              - client_credentials
              id: flytepropeller
              public: false
              redirect_uris:
              - <http://localhost:3846/callback>
              response_types:
              - token
              scopes:
              - all
              - offline
              - access_token
        thirdPartyConfig:
          flyteClient:
            clientId: flytectl
            redirectUri: <http://localhost:53593/callback>
            scopes:
            - offline
            - all
      authorizedUris:
      - <https://localhost:30081>
      - <http://flyteadmin:80>
      - <http://flyteadmin:81>
      - <http://flyteadmin.flyte.svc.cluster.local:80>
      - <http://flyteadmin.flyte.svc.cluster.local:81>
      - <https://flyte.in.cloud.uniphoredev.com>
      - <https://flyte.in.cloud.uniphoredev.com/console>
      userAuth:
        openId:
          baseUrl: '<https://login.microsoftonline.com/><AZ_TENANT_ID_SUBSTITUTED_HERE_>/v2.0'
          clientId: '<AZ_CLIENT_ID_SUBSTITUTED_HERE_>'
          scopes:
          - profile
          - openid
    flyteadmin:
      eventVersion: 2
      metadataStoragePrefix:
      - metadata
      - admin
      metricsScope: 'flyte:'
      profilerPort: 10254
      roleNameKey: <http://iam.amazonaws.com/role|iam.amazonaws.com/role>
      testing:
        host: <http://flyteadmin>
    server:
      grpc:
        port: 8089
      httpPort: 8088
      security:
        allowCors: true
        allowedHeaders:
        - Content-Type
        - flyte-authorization
        allowedOrigins:
        - '*'
        secure: false
        useAuth: true
  storage.yaml: |
    storage:
      type: s3
      container: "uniphore-flyte-dev-in"
      connection:
        auth-type: iam
        region: ap-south-1
      enable-multicontainer: false
      limits:
        maxDownloadMBs: 10
  task_resource_defaults.yaml: |
    task_resources:
      defaults:
        cpu: 1000m
        memory: 1000Mi
        storage: 1000Mi
      limits:
        cpu: 2
        gpu: 1
        memory: 1Gi
        storage: 2000Mi
kind: ConfigMap
metadata:
  name: flyte-admin-base-config
  namespace: flyte

@narrow-king-98655 I'm back and working to reproduce your issue

Thank you @average-finland-92144. Please let me know if you have any updates here.

I'm following the steps in this section of the docs. There's a caveat: the indentation of the

configmap.adminServer.server.security

section is wrong, that section should be under

server

and not at the same level. The rest is fine Completing the steps there leads to no errors on the Pods but the NGINX Ingress controller I'm using doesn't seem to be handling the redirect especially well. The Flyte UI and CLI prompt for authentication but once the redirect is invoked it throws a 502 error. I'm working on it

Thank you very much @average-finland-92144 !! Now we are not seeing this issue.

so seems like this is propeller failing to retrieve the serialized outputs from the S3 bucket There's got to be something wrong with the IRSA setup maybe. I guess you already checked this?

We have done this setup and also have IAM policy as in the documentation. Absence of spark operator would lead this ?

I think if you want to use the Spark operator, there is additional config needed: https://docs.flyte.org/en/latest/deployment/plugins/k8s/index.html

Is there an option to enable verbose on logs ? And, for which pods you need the logs ?

it should be flyteadmin

I dont see any error in default logs

oh that's stable

Yes sure

I see

security

at the same level as server

adminServer:
    server:
      grpc:
        port: 8089
      httpPort: 8088
      security:
        allowCors: true
        allowedHeaders:
        - Content-Type
        allowedOrigins:
        - '*'
        secure: false
        useAuth: true

Based on the base values file, it should be under

server

like in the above snippet

Ahh! My bad. Will check this and confirm once.

from the error message it looks like the scheduler trying to reach flyteadmin over http, can you try with

adminserver.security.secure: true

?