hey! I hope it is a quick one 🙂
we are using this feature (https://docs.flyte.org/en/latest/concepts/data_management.html#raw-data-prefix) trying to return a pandas.DataFrame is the output of a task, but we are hitting this limit (max-output-size-bytes) which is configured to 10MB as default. We are considering increasing this limit but we want to ask if there are any tradeoffs with that.
I'm also wondering why the default value is 10MB. I think that maybe it is very low to work with things like pandas.DataFrame, right? Or maybe I'm misunderstanding something.
09/06/2023, 4:24 PM
wouldn't it be better to offload that to cloud, S3 or GCS than returning a huge dataframe?
I notice an excessive output return significantly slows down Flyte Console, though im not sure if you are using flyte console.