I have a question related to automatic downloads of FlyteFiles/FlyteFolders. I have a large amount of data (~1TB) spread across ~15,000 files stored in GCS. I have a Flyte workflow that, as a first step, downloads these files to be operated on locally. I noticed that the download speed is around 20 MB/s and uses about 1-2 CPU cores. I did some testing, and when I manually invoke

gcloud storage cp -r …

inside of a Flyte workflow, I am able to download the same exact files at approximately 1.7GB/s, and I see usage of ~20 CPU cores (the workflow requests 32 cpu cores). I observe the same behavior when uploading files to GCS as well. How can I still make use of FlyteFiles/FlyteFolders and the automatic download/upload functionality, but get faster download/upload speed?

Hi folks I am attempting to set up azure blob through stow. My values for storage settings looks like so:

storage:
  type: custom
  enableMultiContainer: true
  limits:
    maxDownloadMBs: 500000
    type: custom
  bucketName: "{{ .Values.userSettings.azure.containerName }}"
  custom:
    container: "{{ .Values.userSettings.azure.containerName }}"
    enable-multicontainer: true
    connection: {}
    type: stow
    stow:
      kind: azure
      config:
        account: "{{ .Values.userSettings.azure.storageAccountName }}"
        key: "{{ .Values.userSettings.azure.storageAccountKey }}"

Trying to run the init workflow examples throws:

Traceback (most recent call last):                                                                                                                                                                                                                            │
│   File "/opt/venv/bin/pyflyte-execute", line 8, in <module>                                                                                                                                                                                                   │
│     sys.exit(execute_task_cmd())                                                                                                                                                                                                                              │
│   File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__                                                                                                                                                                          │
│     return self.main(*args, **kwargs)                                                                                                                                                                                                                         │
│   File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main                                                                                                                                                                              ││     rv = self.invoke(ctx)                                                                                                                                                                                                                                     ││   File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke                                                                                                                                                                            ││     return ctx.invoke(self.callback, **ctx.params)                                                                                                                                                                                                            ││   File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke                                                                                                                                                                             ││     return __callback(*args, **kwargs)                                                                                                                                                                                                                        ││   File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 470, in execute_task_cmd                                                                                                                                                      ││     _execute_task(                                                                                                                                                                                                                                            ││   File "/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point                                                                                                                                                 ││     return wrapped(*args, **kwargs)                                                                                                                                                                                                                           ││   File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 348, in _execute_task                                                                                                                                                         ││     _handle_annotated_task(ctx, _task_def, inputs, output_prefix)                                                                                                                                                                                             ││   File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 291, in _handle_annotated_task                                                                                                                                                ││     _dispatch_execute(ctx, task_def, inputs, output_prefix)                                                                                                                                                                                                   ││   File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 160, in _dispatch_execute                                                                                                                                                     ││     ctx.file_access.put_data(ctx.execution_state.engine_dir, output_prefix, is_multipart=True)                                                                                                                                                                ││   File "/opt/venv/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 476, in put_data                                                                                                                                                       ││     raise FlyteAssertion(                                                                                                                                                                                                                                     ││ flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/flyte-z58pqpy5/sandbox/local_flytekit/engine_dir to <abfs://flyte-workflows/metadata/propeller/>...

Original exception: No plugin found for matching protocol of path <abfs://flyte-workflows/metadata/propeller/>...

Are there specific steps to enable/support the abfs protocol, beyond configuring stow as above?

i get

"failed to validate that project [studio] and domain [development] are registered, err: [project [studio] not found]\"}\"\n>>"}

Folks, I have a question for you. We are working on revamping the flytekit configuration system. But, always try very hard not to break anyone. Would it be ok, if we changed flytekit.remote constructor and deprecate some configs. This may break some scripts that use flytekit.remote, but the fix will be very simple. cc @Maarten de Jong / @jeev / @Dylan Wilder

Hi there! I'm trying to deploy Flyte in Azure and when I try to run my first workflow from the examples repo, I get this error:

$ pyflyte run --project flytesnacks --domain development --remote basics/hello_world.py my_wf

Failed with Exception Code: USER:ValueError
Value error!  Received: 400. Request to send data <azure blob storage URI> failed.

Has anyone seen this before? Is it

flytekit

that tries to upload to blob storage or is this error coming from

flyteadmin

? I haven't set up any kind of authentication yet, so that could be related as well.

flyteadmin

also doesn't show any logs even with a high log level Thanks in advance!

Hi I am trying to return a spark data frame but having this issue _user_exceptions.FlyteAssertion( flytekit.exceptions.user.FlyteAssertion: When calling tasks, only keyword args are supported. Aborting execution as detected 1 positional args (FlyteSchema.__class_getitem__.<locals>._TypedSchema(remote_path='/tmp/flyte/20220322_204000/raw/82eaa7c6b77680e044d794beaf4e01cb'),) I have defined column names with their data types as my_schema = FlyteSchema[kwtypes(ID=str,START_T=datetime.datetime,STOP=datetime.datetime,PATIENT=str,ENCOUNTER=str,CODE=str,intDESCRIPTION=str,REASONCODE=int,REASONDESCRIPTION=str)]. Following this https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/k8s_spark/dataframe_passing.html guide

Can anyone help me with flytectl? I am new to flyte and I just deployed flyte sandbox using helm on AWS EKS with ingress. I am wondering how to configure flytectl - especially host part.

Hi. First time trying to deploy Flyte on AWS following these instructions https://docs.flyte.org/en/latest/deployment/aws/manual.html After installing the Helm chart and verifying that the pods are running, I fail to get an address for my hosts:

$ kubectl get ingress -n flyte
NAME              CLASS    HOSTS   ADDRESS   PORTS   AGE
flyte-core        <none>   *                 80      26m
flyte-core-grpc   <none>   *                 80      26m

Describing the ingress shows this

kubectl describe ingress -n flyte
Name:             flyte-core
Namespace:        flyte
Address:          
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /*               ssl-redirect:use-annotation (<error: endpoints "ssl-redirect" not found>)
              /console         flyteconsole:80 (192.168.103.57:8080,192.168.128.109:8080)
              /console/*       flyteconsole:80 (192.168.103.57:8080,192.168.128.109:8080)
              /api             flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /api/*           flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /healthcheck     flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /v1/*            flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /.well-known     flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /.well-known/*   flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /login           flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /login/*         flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /logout          flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /logout/*        flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /callback        flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /callback/*      flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /me              flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /config          flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /config/*        flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /oauth2          flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
              /oauth2/*        flyteadmin:80 (192.168.110.232:8088,192.168.152.242:8088)
Annotations:  <http://alb.ingress.kubernetes.io/actions.ssl-redirect|alb.ingress.kubernetes.io/actions.ssl-redirect>:
                {"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}
              <http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: arn:aws:acm:us-west-2:582526512915:certificate/6c75c8f4-04a1-4aa7-81fa-59c7241e52ba
              <http://alb.ingress.kubernetes.io/group.name|alb.ingress.kubernetes.io/group.name>: flyte
              <http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>: [{"HTTP": 80}, {"HTTPS":443}]
              <http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>: internet-facing
              <http://alb.ingress.kubernetes.io/tags|alb.ingress.kubernetes.io/tags>: service_instance=production
              <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: alb
              <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
              <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
              <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
Events:
  Type     Reason             Age                  From     Message
  ----     ------             ----                 ----     -------
  Warning  FailedDeployModel  5m1s (x18 over 26m)  ingress  Failed deploy model due to InvalidParameter: 1 validation error(s) found.
- minimum field value of 1, CreateTargetGroupInput.Port.


Name:             flyte-core-grpc
Namespace:        flyte
Address:          
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /flyteidl.service.AdminService            flyteadmin:81 (192.168.110.232:8089,192.168.152.242:8089)
              /flyteidl.service.AdminService/*          flyteadmin:81 (192.168.110.232:8089,192.168.152.242:8089)
              /flyteidl.service.DataProxyService        flyteadmin:81 (192.168.110.232:8089,192.168.152.242:8089)
              /flyteidl.service.DataProxyService/*      flyteadmin:81 (192.168.110.232:8089,192.168.152.242:8089)
              /flyteidl.service.AuthMetadataService     flyteadmin:81 (192.168.110.232:8089,192.168.152.242:8089)
              /flyteidl.service.AuthMetadataService/*   flyteadmin:81 (192.168.110.232:8089,192.168.152.242:8089)
              /flyteidl.service.IdentityService         flyteadmin:81 (192.168.110.232:8089,192.168.152.242:8089)
              /flyteidl.service.IdentityService/*       flyteadmin:81 (192.168.110.232:8089,192.168.152.242:8089)
              /grpc.health.v1.Health                    flyteadmin:81 (192.168.110.232:8089,192.168.152.242:8089)
              /grpc.health.v1.Health/*                  flyteadmin:81 (192.168.110.232:8089,192.168.152.242:8089)
Annotations:  <http://alb.ingress.kubernetes.io/actions.ssl-redirect|alb.ingress.kubernetes.io/actions.ssl-redirect>:
                {"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}
              <http://alb.ingress.kubernetes.io/backend-protocol-version|alb.ingress.kubernetes.io/backend-protocol-version>: GRPC
              <http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: arn:aws:acm:us-west-2:582526512915:certificate/6c75c8f4-04a1-4aa7-81fa-59c7241e52ba
              <http://alb.ingress.kubernetes.io/group.name|alb.ingress.kubernetes.io/group.name>: flyte
              <http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>: [{"HTTP": 80}, {"HTTPS":443}]
              <http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>: internet-facing
              <http://alb.ingress.kubernetes.io/tags|alb.ingress.kubernetes.io/tags>: service_instance=production
              <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: alb
              <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
              <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
              <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
              <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
Events:
  Type     Reason             Age                  From     Message
  ----     ------             ----                 ----     -------
  Warning  FailedDeployModel  5m4s (x19 over 27m)  ingress  Failed deploy model due to InvalidParameter: 1 validation error(s) found.
- minimum field value of 1, CreateTargetGroupInput.Port.

Any ideas how to best debug?

Hi I would like to set an environment variable for every Pod that gets scheduled to k8s. What is the preferred way to do this? Specifically I’m trying to set the following for datadog metrics

env:
    - name: DD_AGENT_HOST
      valueFrom:
          fieldRef:
              fieldPath: status.hostIP

Hey everyone, I'm trying the single-cluster deployment of flyte on EKS using the AWS CDK (constrained by work). I'm following "flyte: the hard way". Has anyone done this before? I'll put details of where I'm stuck in the thread