Klemens Kasseroller07/11/2023, 3:08 PM
It looks to me, as if the output (or actually input, since dumping the output to s3 worked fine) using native python literals cannot exceed 2MB after serialization, is that correct? I am trying to pass a list of thousands of paths from one task to another, which apparently takes about 9MB. Is it possible to increase this limit or perhaps the best practice in such a scenario would be to use a Dataframe? Thanks
failed at Node[n4]. BindingResolutionError: Error binding Var [wf].[inp], caused by: failed at Node[n2]. CausedByError: Failed to GetPrevious data from outputDir [<s3://flytemeta/metadata/propeller/ohli-training-development-a9czrgtghc5dzmxhdflf/n2/data/0/outputs.pb>], caused by: path:<s3://flytemeta/metadata/propeller/ohli-training-development-a9czrgtghc5dzmxhdflf/n2/data/0/outputs.pb>: [LIMIT_EXCEEDED] limit exceeded. 8mb > 2mb.
Bernhard Stadlbauer07/11/2023, 3:11 PM
Dan Rammer (hamersaw)07/11/2023, 3:21 PM
Klemens Kasseroller07/11/2023, 3:39 PM
Thomas Blom09/25/2023, 8:03 PM
value. But the file referenced seems to indicate a 10mb default, where my error message (and the OP above) indicates 2mb as the limit, so is this in fact the correct value to change?
Workflow[io-erisyon-people-thomas:development:plaster.genv2.generators.survey.survey_workflow] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: [system] unable to read futures file, maybe corrupted, caused by: [system] Failed to read futures protobuf file., caused by: path:<s3://informatics-flytes3bucket-2133a04/metadata/propeller/io-erisyon-people-thomas-development-f3a22869c5041400ca49/n1/data/0/futures.pb>: [LIMIT_EXCEEDED] limit exceeded. 24mb > 2mb.
Dan Rammer (hamersaw)09/28/2023, 1:36 PM
file in a dynamic workflow is the compiled workflow. So this error may not have anything to do with the result sizes passed between tasks, but rather the overall size of the dynamic. Do you know how many nodes are included in the dynamic workflow? There is different configuration for reading this data, hence the 2MB limit. I think it is being set as maxDownloadMBs (the example has 10MB, but check your configuration please).
Thomas Blom09/28/2023, 3:17 PM
Dan Rammer (hamersaw)09/29/2023, 2:36 PM
file is stored in the blobstore. Basically, in a dynamic flytekit will execute the task and compile the workflow dynamically, then propeller notices the existence of a
file and continues in executing the subnodes for the dynamic. In my local example (below):
I execute on a local cluster with
@task def square(n: int) -> int: return n * n @dynamic def dynamic_parallel_square(a: int, count: int) -> int: for i in range(count): square(n=a) return a @workflow def dynamic_parallel_square_wf(a: int, count: int) -> int: return dynamic_parallel_square(a=a, count=count)
The location of the
(.venv) hamersaw@ragnarok:~/Development/flytetest$ pyflyte run --remote dynamic.py dynamic_parallel_square_wf --a=2 --count=3` Running Execution on [Remote]. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ [✔] Go to <http://localhost:30081/console/projects/flytesnacks/domains/development/executions/f79eae60587bb4140897> to see execution in the console.
in my minio - read as the
execution and data for the first attempt on node
. I read this locally and its only 1.1KB. You can use the
utility (should be installed with pyflyte, but will be deprecated relatively soon) to parse the protobuf file and print json with a command similar to:
This uses flyteidl to print the
(.venv) hamersaw@ragnarok:~/Development/flytetest$ flyte-cli parse-proto --filename ~/Downloads/futures.pb --proto_class flyteidl.core.dynamic_job_pb2.DynamicJobSpec
located here. This should help identify exactly what is being stored for the dynamic.
Thomas Blom09/29/2023, 8:09 PM
We are not changing much of anything from the flyte helm-chart defaults, and per the link you provided, this value would seem to default to 10mb.
[LIMIT_EXCEEDED] limit exceeded. 24mb > 2mb.
Dan Rammer (hamersaw)10/02/2023, 11:26 AM
Thomas Blom10/02/2023, 11:28 AM