Can someone point me to the place(s) in the EKS helm chart that I need to updated to get around this limit?

status = StatusCode.INVALID_ARGUMENT
details = "Requested CPU limit [15] is greater than current limit set in the platform configuration [2]. Please contact Flyte Admins to change these limits or consult the configuration"`

We tend to have some pretty large fan outs and occasionally get hit with the gRPC size limit and wondering if we could think about strategies to prevent users hitting this too. In the most recent case, I had a dynamic that has a list of 4238 items, and will process a task for each item. Each item in the list only has 2 attributes (dataset_id, raw_data_path), but it still exceeds gRPC limits. I'm wondering if it could be possible for Flyte to automatically see that the list is getting large, and serialize to file to prevent gRPC issues (and potentially do this for all collection types)

Folks join the Community Sync today - https://zoom.us/j/93875115830?pwd=YWZWOHl1ODRRVjhjVkxSV0pmZkJaZz09 - happening now

very newbie question: did anyone run into this issue when running the tutorial on the local demo cluster?

Anyone managed to get flyte onto an AWS EKS cluster. I have some questions regarding what needs to be done. I’m following the single cluster instructions, and its basically getting stuck.

Hi a quick question on scaling flyte - when you specify a large amount of resources (e.g. 100 cpus), does flyte scale the nodes in the k8s cluster as well or does that have to be manually handled?

Hey folks.. quick question.. can ContainerTasks be run using the map_task construct?

Hi Flyte community, Is the recommended approach for executing Flyte workflows from Java code using https://github.com/flyteorg/flytekit-java/blob/5bdf81b14911285aa9a15f631a1e84fb74c4755a/jflyte/src/main/java/org/flyte/jflyte/FlyteAdminClient.java or calling the CreateExecute operation directly via GRPC stubs generated from https://mvnrepository.com/artifact/org.flyte/flyteidl-protos?

Hello all I'm getting this error when trying to create a task with custom object:

ValueError: Only generic univariate typing.List[T] type is supported.

I tried to use the example in doc, https://flyte--943.org.readthedocs.build/projects/cookbook/en/943/auto/core/type_system/custom_objects.html

Is it possible for

pyflyte register

to pick up environment variables when I set them like this?

@task(
    task_config=Databricks(databricks_token=os.environ.get("DATABRICKS_TOKEN"))
)
def etl():

Hi folks, I have setup a Flyte based on the Single Cluster Simple Cloud Deployment docs. Deployed a simple workflow based on Getting Started guide. I am getting the following error in the console. Any idea why? Thanks.

[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[f2d93e14fa6164601aad-n0-0] terminated with exit code (137). Reason [OOMKilled]. Message: 
{"asctime": "2023-03-08 13:31:31,597", "name": "flytekit", "levelname": "WARNING", "message": "FlyteSchema is deprecated, use Structured Dataset instead."}
Matplotlib created a temporary config/cache directory at /tmp/matplotlib-mmqveczs because the default path (/home/flytekit/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
.

Hello! I've been setting up Flyte via the single cluster deployment docs and the flyte-binary service fails to start with the following error:

2023/03/08 22:40:38 /go/pkg/mod/gorm.io/driver/postgres@v1.2.3/migrator.go:74
[0.326ms] [rows:1] SELECT CURRENT_DATABASE()
{"json":{"src":"start.go:158"},"level":"panic","msg":"Failed to start Admin, err: database migration failed: ERROR: relation \"description_entities\" does not exist (SQLSTATE 42P01)","ts":"2023-03-08T22:40:38Z"}
panic: (*logrus.Entry) 0xc000aa8460

This is into a fresh RDS database. The migration runs up through

2021-08-05-schedulable_entities_snapshot

verified via

select * from migrations order by id desc limit 1;

. Any help is appreciated!

Hi, While running pyflyte with java 19 from my local pc I'm getting java error. Can anyone tell me whether flyte is java 19 compatible?

@Kevin Su @Samhita Alla @Ketan (kumare3) - We are getting the below error while executing Spark task. Could you please help us An error occurred while calling o125.collectToPython. : java.lang.OutOfMemoryError: Java heap space at org.apache.spark.sql.execution.SparkPlan$$anon$1._next(SparkPlan.scala:391) at org.apache.spark.sql.execution.SparkPlan$$anon$1.getNext(SparkPlan.scala:402) at org.apache.spark.sql.execution.SparkPlan$$anon$1.getNext(SparkPlan.scala:388) at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73) at scala.collection.Iterator.foreach(Iterator.scala:943) at scala.collection.Iterator.foreach$(Iterator.scala:943) at org.apache.spark.util.NextIterator.foreach(NextIterator.scala:21) at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeCollect$1(SparkPlan.scala:425) at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeCollect$1$adapted(SparkPlan.scala:424) at org.apache.spark.sql.execution.SparkPlan$$Lambda$2872/0x0000000841442840.apply(Unknown Source) at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198) at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:424) at org.apache.spark.sql.Dataset.$anonfun$collectToPython$1(Dataset.scala:3688) at org.apache.spark.sql.Dataset$$Lambda$2466/0x0000000841284840.apply(Unknown Source) at org.apache.spark.sql.Dataset.$anonfun$withAction$2(Dataset.scala:3858) at org.apache.spark.sql.Dataset$$Lambda$2735/0x00000008413bdc40.apply(Unknown Source) at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:510) at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3856) at org.apache.spark.sql.Dataset$$Lambda$2467/0x0000000841285440.apply(Unknown Source) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109) at org.apache.spark.sql.execution.SQLExecution$$$Lambda$2475/0x0000000841294840.apply(Unknown Source) at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95) at org.apache.spark.sql.execution.SQLExecution$$$Lambda$2468/0x0000000841290040.apply(Unknown Source) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64) at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3856) at org.apache.spark.sql.Dataset.collectToPython(Dataset.scala:3685) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) User error.

Is anyone aware of a way to modify the cache/version of a task I imported from another module, potentially not controlled by me?

Node.with_overrides

allows me to e.g. change the task resources or config but not its cache.

Hi everyone! 👋 I’m attempting to register an imperative workflow for remote execution, using a bazel build environment, and I’m seeing a failure I don’t completely understand (MWE and error in the thread)

Hi community, is anyone using mpi-operator? I plan to do a upgrade on flyteplugins to mpi-operator v2, and want to seek for some feedbacks! cc @Ketan (kumare3) can you point to some POCs? Thanks!

Hi, we’re running a bunch of large jobs each with thousands of tasks, and noticed our Postgres database was at 100% CPU for many hours. DB only has 4 vCPU and 15GB of memory at the moment. Saw many of these in the datacatalog log as well

"textPayload": "2023/03/07 15:38:00 \u001b[32m/go/src/github.com/flyteorg/datacatalog/pkg/repositories/gormimpl/artifact.go:64 \u001b[33mSLOW SQL >= 200ms",

Do you have a recommended size for postgres for large production workloads? I’m also going through this doc to make sure we’re following best practices.

Hi I just deployed a single cluster on AWS EKS, I'm trying to test-run a workflow on it but I ran into this error:

Original exception: Value error!  Received: 403. Request to send data <https://flytebucket1.s3.us-east-2.amazonaws.com/flytesnacks/development/YYHE6OSLHL2BT5PSBO3SQXDVSQ%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIASDZ4UU6UG2HQ42NX%2F20230309%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20230309T200840Z&X-Amz-Expires=36>

Pretty obvious that it is a problem with S3 but I'm not entirely sure how to go about debugging and fixing this

Hi all - I’m testing some work locally via

flytectl demo

and it’d be really helpful to pull in some files from S3. Is this possible? I’m getting 403 forbidden errors when I run a pipeline that interns such a file.

Hi there, is there a database diagram somewhere explaining the structure of flyte metadata for flyteadmin and datacatalog? I'm trying to get a better handle on how flyte uses these and in particular what data is stored in it (with regards to task outputs for example). Are inputs and outputs from tasks ever stored directly in the database or do they reside only in the backend storage (like s3)?

Hello, I've been working with simple "hello world" pyflyte workflow on a Kubernetes cluster and I've noticed that a task is taking over 90 seconds to run, which is much longer than expected. I'm using the default flyte image on an Intel Mac and running Kubernetes through Docker Desktop. Strangely, running the same flyte workflow locally (i.e., without Kubernetes) only takes a few seconds. Do you have any insights into what could be causing this delay?

hi does project name have to be unique? i’ve tried creating two projects with the same name, and when using reference workflows that references a launchplan in one of the two projects with the same name it throws an error. Is only project id unique right now?

Hi, I am trying to run the following command to build local env:

git clone <https://github.com/flyteorg/flyte.git> 
cd flyte
sudo make compile 
flyte start --config flyte_local.yaml

But after I ran

sudo make compile

to build flyte I got following error. Wondering if I can get some hints on this(It previously works, not sure why it fails now ). Thank you in advance.

root@ip-172-31-7-58:/home/ubuntu/flyte# make compile
script/get_flyteconsole_dist.sh
++ uname -m
+ ARCH=x86_64
+ case ${ARCH} in
+ IMAGE_ARCH=amd64
+ FLYTECONSOLE_IMAGE=<http://ghcr.io/flyteorg/flyteconsole:latest|ghcr.io/flyteorg/flyteconsole:latest>
++ docker manifest inspect --verbose <http://ghcr.io/flyteorg/flyteconsole:latest|ghcr.io/flyteorg/flyteconsole:latest>
++ jq --arg IMAGE_ARCH amd64 --raw-output '.[] | select(.Descriptor.platform.architecture == $IMAGE_ARCH) | .Descriptor.digest'
+ IMAGE_DIGEST=sha256:37cb40ebac15ea7b9aa612a1ac16779079e18e316274dac701383bf13edbf711
+ '[' -f cmd/single/dist/.digest ']'
++ docker create <http://ghcr.io/flyteorg/flyteconsole:latest|ghcr.io/flyteorg/flyteconsole:latest>
+ CONTAINER_ID=f91fd9eca41e682c61b25bd6db42ba02205fd63005893e1ab065fb1cb6549729
+ trap 'docker rm -f ${CONTAINER_ID}' EXIT
+ rm -rf cmd/single/dist
+ docker cp f91fd9eca41e682c61b25bd6db42ba02205fd63005893e1ab065fb1cb6549729:/app/dist cmd/single/
Error response from daemon: Could not find the file /app/dist in container f91fd9eca41e682c61b25bd6db42ba02205fd63005893e1ab065fb1cb6549729
+ docker rm -f f91fd9eca41e682c61b25bd6db42ba02205fd63005893e1ab065fb1cb6549729
f91fd9eca41e682c61b25bd6db42ba02205fd63005893e1ab065fb1cb6549729
make: *** [Makefile:17: cmd/single/dist] Error 1

Do people have a design pattern they like specifying task variants at execution time in a dynamic workflow? As a concrete example, say you have an evaluation workflow and a number of metric functions you want to run within the workflow, specified by the user at execution time. You have several options: You could handle the mapping from dynamic workflow inputs to logic outside of flyte (i.e. pass in a function string name and do a dynamic import within the task, or define an enum with all the task functions, and pass that in to the dynamic workflow) but this feels like a bit of a hat-on-a-hat given that a big part of flyte's functionality is a "logic" registry. I think the other option is passing in a task function directly, and serializing via the key information of the task (name, domain, project, version), then using

reference_task

to deserialize within the dynamic workflow. Does this make sense? I would do this by writing a custom

TypeTransformer

for a

PythonFunctionTask

but I'm wondering if someone has already been down this road and has simpler suggestions?

Hi, I'm trying out the

wait_for_input

gate node with a dummy workflow:

from datetime import timedelta

from flytekit import wait_for_input, workflow, task

@task
def list_folders(additional_choice: str) -> list[str]:
    choices = ['a', 'b', 'c', additional_choice]
    print(f'Choices: {choices}')
    return choices


@task
def chosen_folder(choice: str) -> str:
    print(f'chose {choice}')
    return choice


@workflow
def gating_demo(additional_choice: str):
    choices = list_folders(additional_choice=additional_choice)
    choice = wait_for_input("choose-folder", timeout=timedelta(hours=1), expected_type=str)
    chosen_folder(choice=choice)
    choices >> choice

It works locally, but remotely I get an input modal that doesn't allow me to input anything, complaining that:

The requested item could not be found

. Am I doing something wrong? The input doesn't seem to be required (no asterisk) but skipping input also doesn't work (same error).

Hi there! Do we have any options to increase limits?

failed at Node[n1]. BindingResolutionError: Error binding Var [wf].[chunk_list], caused by: failed at Node[n0]. CausedByError: Failed to GetPrevious data from outputDir [<s3://flyte-data/metadata/propeller/flytesnacks-development-a52pb87sdr8blxsbzgpp/n3/data/0/n0/0/outputs.pb>], caused by: path:<s3://flyte-data/metadata/propeller/flytesnacks-development-a52pb87sdr8blxsbzgpp/n3/data/0/n0/0/outputs.pb>: [LIMIT_EXCEEDED] limit exceeded. 5mb > 2mb.

I've been seeing some interesting behavior on the task decorator when working with a local cluster through Docker Desktop kubernetes on an intel MacOS. I've got a workflow that runs perfectly fine with local

pyflyte run

, but it's failing when I try to set these two arguments in the task decorator:

@task(_secret_requests_=[*AWS_SECRETS], _limits_=Resources(_mem_="1500Mi"))

and then run

--remote

. Either argument works by itself, but when they're both present I get the following error:

Request rejected by the API, due to Invalid input.
	Reason: 
	Input Request: {
  "id": {
    "resourceType": "TASK",
    "project": "REDACTED",
    "domain": "development",
    "name": "cmi_orchestration.tasks.data_generation.get_data_gen_params",
    "version": "FwNcirU-K58BewCeTAtVog=="
  },
  "spec": {
    "template": {
      "id": {
        "resourceType": "TASK",
        "project": "REDACTED",
        "domain": "development",
        "name": "tasks.data_generation.get_data_gen_params",
        "version": "FwNcirU-K58BewCeTAtVog=="
      },
      "type": "python-task",
      "metadata": {
        "runtime": {
          "type": "FLYTE_SDK",
          "version": "1.4.1",
          "flavor": "python"
        },
        "retries": {}
      },
      "interface": {
        "inputs": {
          "variables": {
            "input_location": {
              "type": {
                "simple": "STRING"
              },
              "description": "input_location"
            }
          }
        },
        "outputs": {
          "variables": {
            "o0": {
              "type": {
                "simple": "STRING"
              },
              "description": "o0"
            }
          }
        }
      },
      "container": {
        "image": "localhost:32112/REDACTED",
        "args": [
          "pyflyte-fast-execute",
          "--additional-distribution",
          "<s3://my-s3-bucket/cmi/development/WM5JKLAFUOMF4F7AUZ72CWHATI======/scriptmode.tar.gz>",
          "--dest-dir",
          "/root",
          "--",
          "pyflyte-execute",
          "--inputs",
          "{{.input}}",
          "--output-prefix",
          "{{.outputPrefix}}",
          "--raw-output-data-prefix",
          "{{.rawOutputDataPrefix}}",
          "--checkpoint-path",
          "{{.checkpointOutputPrefix}}",
          "--prev-checkpoint",
          "{{.prevCheckpointPrefix}}",
          "--resolver",
          "flytekit.core.python_auto_container.default_task_resolver",
          "--",
          "task-module",
          "tasks.data_generation",
          "task-name",
          "get_data_gen_params"
        ],
        "resources": {
          "limits": [
            {
              "name": "MEMORY",
              "value": "1500Mi"
            }
          ]
        }
      },
      "securityContext": {
        "secrets": [
          {
            "group": "aws-creds",
            "key": "AWS_ACCESS_KEY_ID",
            "mountRequirement": "ENV_VAR"
          },
          {
            "group": "aws-creds",
            "key": "AWS_SECRET_ACCESS_KEY",
            "mountRequirement": "ENV_VAR"
          },
          {
            "group": "aws-creds",
            "key": "AWS_REGION",
            "mountRequirement": "ENV_VAR"
          }
        ]
      }
    },
    "description": {
      "longDescription": {
        "format": "DESCRIPTION_FORMAT_RST"
      }
    }
  }
}

Hi! Is it possible to fix zooming in a large graph for future releases? Currently, if the workflow has a complex structure maximal zoom cannot fit the entire graph and viewing is very challenging. Screenshot represents maximal available information.

Hi there, if one wanted to temporarily disable the ability for flyte to run new executions (scheduled or manual) via configuration on a deployment, is there a way to accomplish that?