tracing the code, I see: 1. error message emitted here: https://github.com/flyteorg/flyte/blob/aaf6fecb36653e9b57d54fdcc5221731ba82cff5/flytepropeller/pkg/controller/nodes/task/handler.go#L853 2. error defined here: https://github.com/flyteorg/flyte/blob/aaf6fecb36653e9b57d54fdcc5221731ba82cff5/fl[…]s/go/tasks/pluginmachinery/ioutils/remote_file_output_reader.go 3. max payload size config retrieved here: https://github.com/flyteorg/flyte/blob/aaf6fecb36653e9b57d54fdcc5221731ba82cff5/fl[…]s/go/tasks/pluginmachinery/ioutils/remote_file_output_reader.go so the config is: • https://github.com/flyteorg/flyte/blob/master/docs/deployment/configuration/generated/flyteadmin_config.rst#max-output-size-bytes-int64 • https://github.com/flyteorg/flyte/blob/master/docs/deployment/configuration/generated/flyteadmin_config.rst#maxdownloadmbs-int64

related prs: • https://github.com/flyteorg/flyte/pull/4852 • https://github.com/flyteorg/flyte/pull/5092/

why is the default

maxDownloadMBs

set so low (2 MBs)? what are the implications of increasing this to be several hundred gigabytes?

only recommendation I could find online is:

Yes check out propeller config. But preferably offload large lists etc

https://discuss.flyte.org/t/9888328/hi-there-do-we-have-any-options-to-increase-limits-failed-at?t=v2 not sure I understand what offload means in this context. is it suggesting not to return large amounts of data from tasks that are stored in the intermediate storage?

the user that ran into this error here is using

map_task

like so:

# 3. For each day's DailyDataPayload, process them concurrently.
list_of_list_of_results_promise = map_task(process_day_level_files_wf, concurrency=10)(
    daily_payload=daily_payloads_promise, # Pass the list of DailyDataPayload objects
    action_to_find=repeated_actions_promise
)

and the type definition of the task being mapped (

process_day_level_files_wf

) is:

@task(container_image=custom_image)
def process_day_level_files_wf(
    daily_payload: DailyDataPayload,
    action_to_find: str
) -> List[SingleFileResult]:

where

SingleFileResult

is a dataclass defined as:

@dataclass
class SingleFileResult:
    daily_prefix_for_grouping: str
    matched_json_objects: List[Any]

I'm trying to read through this article but I still don't quite understand what needs to be tweaked here: https://www.union.ai/docs/flyte/user-guide/data-input-output/task-input-and-output/#metadata-and-raw-data

I see the workflow is using lots of

List[str]

and

List[int]

-- perhaps that is getting saved as metadata instead of a pointer to raw data?

Can someone point me to the code where we determine whether a type is considered primitive or complex?: > Primitive values (

int

,

str

, etc.) are stored directly in the metadata store, while complex data objects (

pandas.DataFrame

,

FlyteFile

, etc.) are stored by reference, with the reference pointer in the metadata store and the actual data in the raw data store.