I have deployed the Flyte Sandbox to a remote machine -- UI is accessible and everything is up. I have set the endpoint in ~/.flyte/config.yaml. When I run a workflow using PyFlyte and a custom container in a registry local to my machine, I get the following error:

flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/tmpby0anszu/script_mode.tar.gz to <http://localhost:30084/my-s3-bucket/ff/flytesnacks/development/MOAHDD5B6MWAWQCHZQVZWQX6UU%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20220715%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220715T180620Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=d807a4f73adf126a579aa7dcbe1800e8654559f37c8728500b61d20b067d804e> (recursive=False).

Original exception: HTTPConnectionPool(host='localhost', port=30084): Max retries exceeded with url: /my-s3-bucket/ff/flytesnacks/development/MOAHDD5B6MWAWQCHZQVZWQX6UU%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20220715%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220715T180620Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=d807a4f73adf126a579aa7dcbe1800e8654559f37c8728500b61d20b067d804e (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f440c5cd910>: Failed to establish a new connection: [Errno 111] Connection refused'))

Question about nested types: I have a task that receives

List[Dict[str, pd.DataFrame]]

and get the following error:

File ".../flytekit/core/type_engine.py", line 810, in to_python_value
    return [TypeEngine.to_python_value(ctx, x, st) for x in lv.collection.literals]
AttributeError: 'NoneType' object has no attribute 'literals'

Flytekit version 0.31.0 - more context in comments

Hi Everyone I am trying to enable authentication and authorization in flyte . and it is external authentication by Microsoft Azure. I am following this documentation of flyte https://docs.flyte.org/en/latest/deployment/configuration/auth_setup.html I am using flyte on the top of kubernetes (Single Node Cluster). where minio and postgress is already configured for flyte namespace. Although there was an error came up previously regarding securecookie is too long. So I followed this patch version of flyte https://github.com/flyteorg/flyte/pull/4863 I use this repos flyte-binary to deploy our flyte-binary pod on our cluster. But now it is giving me following error in flyte-binary pod logs: I am using helm to deloy flyte-binary. I am also adding values.yaml file . Please let me know if there is any configuration error. Or anything i can do to solve this error. I have to solve it as soon as possible. values.yaml

# configuration Specify configuration for Flyte
configuration:
  # database Specify configuration for Flyte's database connection
  database:
    username: <username>
    password: <password>
    host: postgres.flyte.svc.cluster.local
    port: 5432
    dbname: flyteadmin
    options: sslmode=disable
  # storage Specify configuration for object store
  storage:
    type: minio
    metadataContainer: "flyte-bucket"
    userDataContainer: "flyte-bucket"
    provider: s3
    providerConfig:
      # s3 Provider configuration for S3 object store
      s3:
        region: "us-east-1"
        disableSSL: true
        secure: false
        v2Signing: false
        endpoint: "<http://minio.flyte.svc.cluster.local:9200>"
        authType: "accesskey"
        accessKey: <accesskey>
        secretKey: <secret-key>
      
  # logging Specify configuration for logs emitted by Flyte
  logging:
    # level Set the log level
    level: 6

  # auth Specify configuration for Flyte authentication
  auth:
    enabled: true
    oidc:
      baseUrl: <https://login.microsoftonline.com/><tenant-id>/v2.0
      clientId: <client-id>
      clientSecret: <client-secret>

    internal:
      clientSecret: <client-secret>
      clientSecretHash: <client-secret-hash>
    authorizedUris:
    - <https://mlflyte.accounts.intern>
  
  inline:
    auth:
      appAuth:
        authServerType: External
        externalAuthServer:
          baseUrl: <https://login.microsoftonline.com/><tenant-id>/v2.0/
          metadataUrl: .well-known/openid-configuration
          AllowedAudience: 
            - api://<client-id>            
        thirdPartyConfig:
          flyteClient:
              clientId: <client-id>
              redirectUri: <http://localhost:53593/callback>
              scopes:
                - profile
                - openid
                - email
                - offline_access
      userAuth:
        openId:
          baseUrl: <https://login.microsoftonline.com/><tenant-id>/v2.0
          scopes:
            - profile
            - openid
            - email
            - offline_access
          clientId: <client-id>

      plugins:
        k8s:
          inject-finalizer: true
          default-env-vars:
            - FLYTE_AWS_ENDPOINT: "<http://minio.flyte.svc.cluster.local:9200>"
            - FLYTE_AWS_ACCESS_KEY_ID: <access-key>
            - FLYTE_AWS_SECRET_ACCESS_KEY: <secret-key> #Use the same value as the MINIO_ROOT_PASSWORD
      task_resources:
        defaults: 
          cpu: 100m
          memory: 500Mi #change default requested resources and limits to fit your needs
        limits:
          memory: 5Gi
      storage:
        cache:
          max_size_mbs: 100
          target_gc_percent: 100

# deployment Configure Flyte deployment specification
deployment:
  # image Configure image to use for Flyte
  image:
    repository: <repo-link>/flyte-binary # FLYTE_IMAGE
    tag: native # v1.10.6 # FLYTE_TAG
    pullPolicy: IfNotPresent
  # extraEnvVars Array with extra environment variables to add to Flyte
  extraEnvVars:
    - name: HTTP_PROXY
      value: <PROXY-VALUE>
    - name: HTTPS_PROXY
      value: <PROXY-VALUE>
    - name: http_proxy
      value: <PROXY-VALUE>
    - name: https_proxy
      value: <PROXY-VALUE>
    - name: no_proxy
      value: <PROXY-VALUE>
    - name: NO_PROXY
      value: <PROXY-VALUE>
    - name: GODEBUG
      value: "tlsmaxrsasize=16384"
  # extraEnvVarsConfigMap: environment-values
  
# ingress Configure ingress for Flyte
ingress:
  create: true
  host: mlflyte.accounts.intern
  commonAnnotations: 
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
    <http://ingress.kubernetes.io/rewrite-target|ingress.kubernetes.io/rewrite-target>: /
    <http://nginx.ingress.kubernetes.io/proxy-buffer-size|nginx.ingress.kubernetes.io/proxy-buffer-size>: "32k"
    <http://nginx.ingress.kubernetes.io/proxy-read-timeout|nginx.ingress.kubernetes.io/proxy-read-timeout>: "600"
    <http://nginx.ingress.kubernetes.io/proxy-send-timeout|nginx.ingress.kubernetes.io/proxy-send-timeout>: "600"
    <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
    <http://nginx.ingress.kubernetes.io/affinity|nginx.ingress.kubernetes.io/affinity>: "cookie"
    <http://nginx.ingress.kubernetes.io/session-cookie-name|nginx.ingress.kubernetes.io/session-cookie-name>: "flyte_idt"
  # httpAnnotations Add annotations to http ingress resource
  httpAnnotations:
    <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
  # grpcAnnotations Add annotations to grpc ingress resource
  grpcAnnotations:
    <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
  # ingressClassName Ingress class to use with all ingress resources
  ingressClassName: nginx
  # httpIngressClassName Ingress class to use with all http ingress resource. Overrides `ingressClassName`
  httpIngressClassName: ""
  # grpcIngressClassName Ingress class to use with all grpc ingress resource. Overrides `ingressClassName`
  grpcIngressClassName: ""
  # tls Add TLS configuration to all ingress resources
  tls:
    - hosts:
        - mlflyte.accounts.intern
      secretName: mlflyte
  # httpTls Add TLS configuration to http ingress resource. Overrides `tls`
  httpTls: []
  # grpcTls Add TLS configuration to grpc ingress resource. Overrides `tls`
  grpcTls: []
  # httpExtraPaths Add extra paths to http ingress rule
  httpExtraPaths:
    prepend: []
    append: []
  # grpcExtraPaths Add extra paths to grpc ingress rule
  grpcExtraPaths:
    prepend: []
    append: []


# serviceAccount Configure Flyte ServiceAccount
serviceAccount:
  # create Create ServiceAccount for Flyte
  create: true
  # name Name of service account
  name: ""
  # labels Add labels to ServiceAccount
  labels: {}
  # annotations Add annotations to ServiceAccount
  annotations: {}
  # imagePullSecrets Secrets to use for fetching images from private registries
  # imagePullSecrets: [{ name: developerkey }]
  imagePullSecrets: [{ name: <pull-image-secret> }]

Hi, does Flyte not support TypedDict?

Failed to convert return value for var o0 for function project.wf_58_366.wf_58.dataset_tylo_credit_card_fraud with error <class 'TypeError'>: TypedDict does not support instance and class checks

What should I use instead if I want to use an alias type which is a dictionary with specific keys?

Hi everyone, Is it possible to run part of a worklow or a list of tasks in a single pod? I would like to use the caching mechanism for tasks but reduce the cost of starting a pod for each task. I have a task that i want to call up thousands of times, but it doesn't require a lot of computing time

Hi, some more questions about Flyte caching. Is there a way to view the remote cache and answer questions like.. • See what tasks have cache entries and what keys/values are being stored? • See how much storage it’s taking up? • How often the cache is hit or missed when a task is called?

Anyone managed to get flyte onto an AWS EKS cluster. I have some questions regarding what needs to be done. I’m following the single cluster instructions, and its basically getting stuck.

Hi, In a model pipeline, the tasks are taking longer time to run than usual. A func. which would take ~2 mins to run is taking 35-40 min when using flyte. When I checked the cloudwatch logs, I see that there is a dormant period in every task which is taking up >98% of run time. In the image attached below, after 18 min (@ 17:27), the actual process is running and it takes seconds to complete. In between the initialization and the actual code run, when I checked the logs, these are available:

flyteadmin-c9cccb94d-fcq5s_flyte_flyteadmin-51bb75904a0581e506a547fc4e434e29b564ec2bcf9d41c343575ffdb29c5bc4

flytepropeller-cbf5cb9f-9kn76_flyte_flytepropeller-b5c4de694b13a1687857b30012166b8218f8ddd20e141f94492b441fa7fdf425

flyteconsole-5c7b456999-jqt42_flyte_flyteconsole-7f461bce1be5bcf45010474a20c7b6602ed0892374d968838ca2ed72f9c1d878

Are these contributing to the increased runtime ?

i’m trying to pull images from a private container image registry by following these instructions and running into a hiccup. we’re using ghcr.io. i have the token for a service account. i can pull images locally (on my laptop) from the private registry. i’m confused about what comes next. a some questions: • the docs say “Use a custom pod template to create a pod. This template is automatically added to every

pod

that Flyte creates.” does that custom pod template get created in the

flytepropeller

deployment? what should be included in that template? im assuming it should specify an image that has a docker config with the auth data. but that image would have to be public? so i think im missing something there. • the docs then say “Update FlytePropeller about the pod created in the previous step.” does that mean to update the

flyte-propeller-config

configmap value

co-pilot.yaml.plugins.k8s.default-pod-template-name

to whatever the

name

value was in the pod template defined in the previous step?

Hey all, I am trying to access AWS services from tasks running in the demo cluster. When I set

AWS_ACCESS_KEY_ID

as environment variable in a task I get the following error (here, I tried it with the example workflow from the documentation, https://docs.flyte.org/en/latest/getting_started/index.html, just adding

os.environ["AWS_ACCESS_KEY_ID"]="secret_value"

to a task). It seems that the environment variable is picked up when artifacts are stored on minio? Does anybody know how to resolve it? What am I missing here?

[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[fc0d281b2d8144efcb5d-n0-0] terminated with exit code (1). Reason [Error]. Message: 
cess exited with error code: 1.  Stderr dump:

b'upload failed: ../tmp/flyte-0ws144nx/sandbox/local_flytekit/engine_dir/error.pb to <s3://my-s3-bucket/metadata/propeller/flytesnacks-development-fc0d281b2d8144efcb5d/n0/data/0/error.pb> An error occurred (AccessDenied) when calling the PutObject operation: Access Denied.\n'
Traceback (most recent call last):
  File "/usr/local/bin/pyflyte-fast-execute", line 8, in <module>
    sys.exit(fast_execute_task_cmd())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/flytekit/bin/entrypoint.py", line 507, in fast_execute_task_cmd
    subprocess.run(cmd, check=True)
  File "/usr/local/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['pyflyte-execute', '--inputs', '<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-fc0d281b2d8144efcb5d/n0/data/inputs.pb>', '--output-prefix', '<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-fc0d281b2d8144efcb5d/n0/data/0>', '--raw-output-data-prefix', '<s3://my-s3-bucket/test/hx/fc0d281b2d8144efcb5d-n0-0>', '--checkpoint-path', '<s3://my-s3-bucket/test/hx/fc0d281b2d8144efcb5d-n0-0/_flytecheckpoints>', '--prev-checkpoint', '""', '--dynamic-addl-distro', '<s3://my-s3-bucket/b6/flytesnacks/development/A7HCCVU2345H3DD7M6S5QIAJ2U======/scriptmode.tar.gz>', '--dynamic-dest-dir', '/root', '--resolver', 'flytekit.core.python_auto_container.default_task_resolver', '--', 'task-module', 'example_workflow', 'task-name', 'generate_normal_df']' returned non-zero exit status 1.
.