Hey all, I'm getting the following error after running 2 map tasks one after the other (with 470~ sub-tasks in each map_task):

BindingResolutionError: Error binding Var [wf].[eeg_data], caused by: failed at Node[n1]. CausedByError: Failed to GetPrevious data from outputDir 
[LIMIT_EXCEEDED] limit exceeded. 2.903926mb > 2mb

the task contain a dataclass input parameter named eeg_data (decorated with mashumaru's DataClassJSONMixin) which seems to be saved in metadata. The metadata from the map_task(which seems to save output metadata from all subtasks in one file) seems to exceed the limit.. How can I increase the limit? I see that it is configured to 2mb in the file flyte/flytestdlib/storage/config.go:

ConfigSection = config.MustRegisterSection(configSectionKey, defaultConfig)
	defaultConfig = &Config{
		Type: TypeS3,
		Limits: LimitsConfig{
			GetLimitMegabytes: 2,
		},
		Connection: ConnectionConfig{
			Region:   "us-east-1",
			AuthType: "iam",
		},
		MultiContainerEnabled: false,
	}
)

@Guy Harel

Set the storage limit to 10MB. This should be fine. Your data set metadata has increased beyond 2MB

@Ketan (kumare3) Thanks!

also a heads up that the storage limit config in propeller (MaxDatasetSizeByte) will be deprecated in the next Flyte release in favor of the GetLimitMegabytes storage config.

Thanks, the relevant one in this case was maxDownloadMBs

oops - I meant to put maxDownloadMBs instead of GetLimitMegabytes for the storage config