white-teacher-47376
07/11/2023, 3:08 PMfailed at Node[n4]. BindingResolutionError: Error binding Var [wf].[inp], caused by: failed at Node[n2]. CausedByError: Failed to GetPrevious data from outputDir [<s3://flytemeta/metadata/propeller/ohli-training-development-a9czrgtghc5dzmxhdflf/n2/data/0/outputs.pb>], caused by: path:<s3://flytemeta/metadata/propeller/ohli-training-development-a9czrgtghc5dzmxhdflf/n2/data/0/outputs.pb>: [LIMIT_EXCEEDED] limit exceeded. 8mb > 2mb.
It looks to me, as if the output (or actually input, since dumping the output to s3 worked fine) using native python literals cannot exceed 2MB after serialization, is that correct?
I am trying to pass a list of thousands of paths from one task to another, which apparently takes about 9MB. Is it possible to increase this limit or perhaps the best practice in such a scenario would be to use a Dataframe?
Thanksagreeable-kitchen-44189
07/11/2023, 3:11 PMhallowed-mouse-14616
07/11/2023, 3:21 PMmax-output-size-bytes
, at the root of the propeller configuraiton.freezing-airport-6809
white-teacher-47376
07/11/2023, 3:39 PMmicroscopic-furniture-57275
09/25/2023, 8:03 PMmax-output-size-bytes
value. But the file referenced seems to indicate a 10mb default, where my error message (and the OP above) indicates 2mb as the limit, so is this in fact the correct value to change?microscopic-furniture-57275
09/25/2023, 10:07 PMWorkflow[io-erisyon-people-thomas:development:plaster.genv2.generators.survey.survey_workflow] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: [system] unable to read futures file, maybe corrupted, caused by: [system] Failed to read futures protobuf file., caused by: path:<s3://informatics-flytes3bucket-2133a04/metadata/propeller/io-erisyon-people-thomas-development-f3a22869c5041400ca49/n1/data/0/futures.pb>: [LIMIT_EXCEEDED] limit exceeded. 24mb > 2mb.
microscopic-furniture-57275
09/25/2023, 10:10 PMhallowed-mouse-14616
09/28/2023, 1:36 PMfutures.pb
file in a dynamic workflow is the compiled workflow. So this error may not have anything to do with the result sizes passed between tasks, but rather the overall size of the dynamic. Do you know how many nodes are included in the dynamic workflow? There is different configuration for reading this data, hence the 2MB limit. I think it is being set as maxDownloadMBs (the example has 10MB, but check your configuration please).microscopic-furniture-57275
09/28/2023, 3:17 PMhallowed-mouse-14616
09/29/2023, 2:36 PMfutures.pb
file is stored in the blobstore. Basically, in a dynamic flytekit will execute the task and compile the workflow dynamically, then propeller notices the existence of a futures.pb
file and continues in executing the subnodes for the dynamic. In my local example (below):
@task
def square(n: int) -> int:
return n * n
@dynamic
def dynamic_parallel_square(a: int, count: int) -> int:
for i in range(count):
square(n=a)
return a
@workflow
def dynamic_parallel_square_wf(a: int, count: int) -> int:
return dynamic_parallel_square(a=a, count=count)
I execute on a local cluster with
(.venv) hamersaw@ragnarok:~/Development/flytetest$ pyflyte run --remote dynamic.py dynamic_parallel_square_wf --a=2 --count=3`
Running Execution on [Remote]. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[✔] Go to <http://localhost:30081/console/projects/flytesnacks/domains/development/executions/f79eae60587bb4140897> to see execution in the console.
The location of the futues.pb
file is my-s3-bucket/metadata/propeller/flytesnacks-development-f79eae60587bb4140897/n0/data/0/futures.pb
in my minio - read as the flytesnacks-developemnt-f...
execution and data for the first attempt on node n0
. I read this locally and its only 1.1KB. You can use the flyte-cli
utility (should be installed with pyflyte, but will be deprecated relatively soon) to parse the protobuf file and print json with a command similar to:
(.venv) hamersaw@ragnarok:~/Development/flytetest$ flyte-cli parse-proto --filename ~/Downloads/futures.pb --proto_class flyteidl.core.dynamic_job_pb2.DynamicJobSpec
This uses flyteidl to print the DynamicJobSpec
located here. This should help identify exactly what is being stored for the dynamic.microscopic-furniture-57275
09/29/2023, 8:09 PMmicroscopic-furniture-57275
09/29/2023, 8:26 PM[LIMIT_EXCEEDED] limit exceeded. 24mb > 2mb.
We are not changing much of anything from the flyte helm-chart defaults, and per the link you provided, this value would seem to default to 10mb.microscopic-furniture-57275
09/29/2023, 8:40 PMhallowed-mouse-14616
10/02/2023, 11:26 AMmicroscopic-furniture-57275
10/02/2023, 11:28 AM