Klemens Kasseroller
07/11/2023, 3:08 PMfailed at Node[n4]. BindingResolutionError: Error binding Var [wf].[inp], caused by: failed at Node[n2]. CausedByError: Failed to GetPrevious data from outputDir [<s3://flytemeta/metadata/propeller/ohli-training-development-a9czrgtghc5dzmxhdflf/n2/data/0/outputs.pb>], caused by: path:<s3://flytemeta/metadata/propeller/ohli-training-development-a9czrgtghc5dzmxhdflf/n2/data/0/outputs.pb>: [LIMIT_EXCEEDED] limit exceeded. 8mb > 2mb.
It looks to me, as if the output (or actually input, since dumping the output to s3 worked fine) using native python literals cannot exceed 2MB after serialization, is that correct?
I am trying to pass a list of thousands of paths from one task to another, which apparently takes about 9MB. Is it possible to increase this limit or perhaps the best practice in such a scenario would be to use a Dataframe?
ThanksBernhard Stadlbauer
07/11/2023, 3:11 PMDan Rammer (hamersaw)
07/11/2023, 3:21 PMmax-output-size-bytes
, at the root of the propeller configuraiton.Ketan (kumare3)
Klemens Kasseroller
07/11/2023, 3:39 PMThomas Blom
09/25/2023, 8:03 PMmax-output-size-bytes
value. But the file referenced seems to indicate a 10mb default, where my error message (and the OP above) indicates 2mb as the limit, so is this in fact the correct value to change?Workflow[io-erisyon-people-thomas:development:plaster.genv2.generators.survey.survey_workflow] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: [system] unable to read futures file, maybe corrupted, caused by: [system] Failed to read futures protobuf file., caused by: path:<s3://informatics-flytes3bucket-2133a04/metadata/propeller/io-erisyon-people-thomas-development-f3a22869c5041400ca49/n1/data/0/futures.pb>: [LIMIT_EXCEEDED] limit exceeded. 24mb > 2mb.
Dan Rammer (hamersaw)
09/28/2023, 1:36 PMfutures.pb
file in a dynamic workflow is the compiled workflow. So this error may not have anything to do with the result sizes passed between tasks, but rather the overall size of the dynamic. Do you know how many nodes are included in the dynamic workflow? There is different configuration for reading this data, hence the 2MB limit. I think it is being set as maxDownloadMBs (the example has 10MB, but check your configuration please).Thomas Blom
09/28/2023, 3:17 PMDan Rammer (hamersaw)
09/29/2023, 2:36 PMfutures.pb
file is stored in the blobstore. Basically, in a dynamic flytekit will execute the task and compile the workflow dynamically, then propeller notices the existence of a futures.pb
file and continues in executing the subnodes for the dynamic. In my local example (below):
@task
def square(n: int) -> int:
return n * n
@dynamic
def dynamic_parallel_square(a: int, count: int) -> int:
for i in range(count):
square(n=a)
return a
@workflow
def dynamic_parallel_square_wf(a: int, count: int) -> int:
return dynamic_parallel_square(a=a, count=count)
I execute on a local cluster with
(.venv) hamersaw@ragnarok:~/Development/flytetest$ pyflyte run --remote dynamic.py dynamic_parallel_square_wf --a=2 --count=3`
Running Execution on [Remote]. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[✔] Go to <http://localhost:30081/console/projects/flytesnacks/domains/development/executions/f79eae60587bb4140897> to see execution in the console.
The location of the futues.pb
file is my-s3-bucket/metadata/propeller/flytesnacks-development-f79eae60587bb4140897/n0/data/0/futures.pb
in my minio - read as the flytesnacks-developemnt-f...
execution and data for the first attempt on node n0
. I read this locally and its only 1.1KB. You can use the flyte-cli
utility (should be installed with pyflyte, but will be deprecated relatively soon) to parse the protobuf file and print json with a command similar to:
(.venv) hamersaw@ragnarok:~/Development/flytetest$ flyte-cli parse-proto --filename ~/Downloads/futures.pb --proto_class flyteidl.core.dynamic_job_pb2.DynamicJobSpec
This uses flyteidl to print the DynamicJobSpec
located here. This should help identify exactly what is being stored for the dynamic.Thomas Blom
09/29/2023, 8:09 PM[LIMIT_EXCEEDED] limit exceeded. 24mb > 2mb.
We are not changing much of anything from the flyte helm-chart defaults, and per the link you provided, this value would seem to default to 10mb.Dan Rammer (hamersaw)
10/02/2023, 11:26 AMThomas Blom
10/02/2023, 11:28 AM