Hello all! I’m deploying flyte to aws eks. I followed the instructions in the docs, and everything seems to be set up. However, the ALB ingress controller is not giving me an address.

$ kubectl -n flyte get ingress
NAME              CLASS    HOSTS   ADDRESS   PORTS   AGE
flyte-core        <none>   *                 80      41m
flyte-core-grpc   <none>   *                 80      41m

This issue was mentioned in the troubleshooting section of the docs, and it suggested running

$ kubectl describe ingress -n flyte
Name:             flyte-core
Labels:           <http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
Namespace:        flyte
Address:          
Ingress Class:    <none>
Default backend:  <default>
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /*               ssl-redirect:use-annotation (<error: endpoints "ssl-redirect" not found>)
              
...

Annotations:  <http://alb.ingress.kubernetes.io/actions.ssl-redirect|alb.ingress.kubernetes.io/actions.ssl-redirect>:
                {"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}
              <http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: arn:aws:acm:us-east-1:752578504353:certificate/7a7065cb-1ffc-418e-8070-bc36fbaff7cb
              <http://alb.ingress.kubernetes.io/group.name|alb.ingress.kubernetes.io/group.name>: flyte
              <http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>: [{"HTTP": 80}, {"HTTPS":443}]
              <http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>: internet-facing
              <http://alb.ingress.kubernetes.io/tags|alb.ingress.kubernetes.io/tags>: service_instance=production
              <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: alb
              <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
              <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
              <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
Events:
  Type     Reason             Age                   From     Message
  ----     ------             ----                  ----     -------
  Warning  FailedDeployModel  3m33s (x20 over 42m)  ingress  Failed deploy model due to InvalidParameter: 1 validation error(s) found.
- minimum field value of 1, CreateTargetGroupInput.Port.


Name:             flyte-core-grpc
Labels:           <http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
Namespace:        flyte
Address:          
Ingress Class:    <none>
Default backend:  <default>
Rules:
  Host        Path  Backends
  ----        ----  --------

  ...

Annotations:  <http://alb.ingress.kubernetes.io/actions.ssl-redirect|alb.ingress.kubernetes.io/actions.ssl-redirect>:
               {"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}

...

Events:
  Type     Reason             Age                   From     Message
  ----     ------             ----                  ----     -------
  Warning  FailedDeployModel  3m36s (x19 over 42m)  ingress  Failed deploy model due to InvalidParameter: 1 validation error(s) found.
- minimum field value of 1, CreateTargetGroupInput.Port.

Could someone point me to the right direction in debugging this? I checked the security groups of EKS cluster and RDS, they were the same. Thanks in advance. Best, Ekku, CTO & Co-Founder @ inven.ai

Hi all, We've started to "upgrade" from flyte-sandbox to flyte-binary and actually I'm facing some challenges. Does someone have a running setup with flyte-binary and SSO with AzureAD/OIDC for FlyteConsole and Flytectl? It seems that FlyteConsole doesn't send a token request and so it doesn't get a JWT token and authorization fails. Here's my values.yaml file:

configuration:
  database:
    host: postgresql.mlops.svc.cluster.local
    dbname: flyteadmindb
    username: flyteuser
    password: "..."
    options: sslmode=disable
  storage:
    type: minio
    metadataContainer: "flyte-container"
    userDataContainer: "flyte-container"
    provider: s3
    providerConfig:
      # s3 Provider configuration for S3 object store
      s3:
        # disableSSL Switch to disable SSL for communicating with S3-compatible service
        disableSSL: true
        # v2Signing Flag to sign requests with v2 signature
        # Useful for s3-compatible blob stores (e.g. minio)
        v2Signing: false
        # endpoint URL of S3-compatible service
        endpoint: <http://mls3api.corp.intern:9000/>
        # authType Type of authentication to use for connecting to S3-compatible service (Supported values: iam, accesskey)
        authType: accesskey
        # accessKey Access key for authenticating with S3-compatible service
        accessKey: "..."
        # secretKey Secret key for authenticating with S3-compatible service
        secretKey: "..."
  logging:
    level: 5
  auth:
    enabled: true
    oidc:
      baseUrl: "<https://login.microsoftonline.com/tenant_id/v2.0>"
      clientId: "..."
      clientSecret: "..."
      scopes:
        - openid
        - email
        - profile
    internal:
      clientSecret: "..."
      clientSecretHash: ".."
    flyteClient:
    # clientId Client ID for Flyte client authentication
      clientId: "..."
      # redirectUri Redirect URI for Flyte client authentication
      redirectUri: "<http://localhost:53593/callback>"
      # scopes Scopes for Flyte client authentication
      scopes:
        - all
    authorizedUris:
    - <https://login.microsoftonline.com/tenant_id/oauth2/v2.0>
    - <https://mlflyte.corp.intern>

  inline:
    plugins:
      k8s:
        inject-finalizer: true
        default-env-vars:
          - AWS_METADATA_SERVICE_TIMEOUT: 5
          - AWS_METADATA_SERVICE_NUM_ATTEMPTS: 20
    storage:
      cache:
        max_size_mbs: 100
        target_gc_percent: 100

serviceAccount:
  create: true
  annotations: {}

ingress:
  create: true
  commonAnnotations:
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
  httpAnnotations:
    <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
  grpcAnnotations:
    <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC

deployment:
  extraEnvVars:
    - name: HTTP_PROXY
      value: "..."
    - name: HTTPS_PROXY
      value:  "..."
    - name: NO_PROXY
      value: "..."
    - name: no_proxy
      value: "..."

Does anyone run flyte on graviton instances / arm64? I am looking at docker images and they seem to be only produced for amd64 (though I could be missing something).

Hello, when I am packaging using

pyflyte package

command (following this guide basically) I am running into a strange issue where the serialized task definitions end up with a resolver path like

--resolver site-packages.flytekit.core.python_auto_container.default_task_resolver

Note the leading

site-packages

. Anyone know why this might be getting added / how to resolve it?

Hello Flyte folks, so I am trying to use Optional type in our workflow, however i have been seeing this error, i have reproduced in a sandbox as well. More 🧵

Hi all, does oauth2 let me limit what projects users can read/execute? Is it possible to give users less than admin access? The docs here https://docs.flyte.org/en/latest/deployment/configuration/auth_setup.html#oauth2-authorization-server seem to insinuate that I can use an external auth server to specify scopes to particular users, but I can't seem to find a list of scopes anywhere in the docs. I'm new to auth flows, so apologies if I'm misguided "looking for oauth scopes in the docs." Any hint in the right direction would be appreciated.

👋 I'm running into an auth problem with flytectl using the ClientSecret (client credentials) method. I've assigned the proper scopes to the oauth2 client and the flyteadmin service isn't complaining about not being able to find the token in the metadata anymore which is nice. However now I'm getting a response to flytectl that I'm missing a scope:

Error: Connection Info: [Endpoint: dns:///flyte.localhost:80, InsecureConnection?: true, AuthMode: ClientSecret]: rpc error: code = Unauthenticated desc = authenticated user doesn't have required scope

LinkerD confirms that it's the flyteadmin container that responds with Unauthenticated. This is quite perplexing and I've verified that the token is sent and it contains all the scopes required:

"scope": "all email offline profile"

I hope people can give me some hints as I've been bashing my head for way to long against this. I'm just trying to have a flytectl register stuff in CI.

Hi how can I access the swagger documentation for flyte rest APIs. I started the sandbox and tried to look at localhost:30081/openapi but could not access

Hello ^_^ Before trying this out myself, has there been any attempts/thoughts on enabling nvidia-acceleration for the single container demo cluster?

When I try to use

with_overrides(pod_template_name="string")

on a NotebookTask, it appears that it doesn't do anything. Is this a Papermill/NotebookTask issue or have I perhaps specified it incorrectly? Setting the

pod_template_name="string

parameter directly on the NotebookTask definition itself works, but not with

with_overrides

. Basically, this works:

nb = NotebookTask(
    pod_template_name="random template name",
)

@dynamic
def function():
    return nb

# No problem

But overriding doesn't work within a dynamic workflow:

@dynamic()
def dynamic_workflow() -> List[float]:

    task_output = nb().with_overrides(
        pod_template_name="string to override"
    )

    return task_output

# Pod template name isn't overriden