@ancient-wolf-19325 hey, I am using minio as my local s3 storage with plenty of PVC under it, but after a while during workflow executing, I am receiving this error: FlyteUploadDataException: SYSTEMUploadDataError error=Failed to put data from /tmp/flyteatfivyar/local_flytekit/engine_dir to s3://flyte/metadata/propeller/my-development-ahclbhngkh57l872khzq/n2/data/0/28/1 (recursive=True). Original exception: [Errno 5] An error occurred (XMinioStorageFull) when calling the PutObject operation: Storage backend has reached its minimum free drive threshold. Please delete a few objects to proceed., cause=[Errno 5] An error occurred (XMinioStorageFull) when calling the PutObject operation: Storage backend has reached its minimum free drive threshold. Please delete a few objects to proceed. what could be the problem? How can I increase the bucket’s size ? (if this is the problem)

The error "XMinioStorageFull" indicates that your MinIO backend has run out of available storage space, hitting its minimum free drive threshold. This is not a Flyte-specific issue, but rather a storage limitation on your MinIO deployment. To resolve this, you need to increase the storage allocated to MinIO, which is typically managed via the PersistentVolume (PV) and PersistentVolumeClaim (PVC) settings in your Kubernetes cluster. To increase the bucket's (MinIO's) storage size, you should update the

storage

section in your MinIO PV and PVC manifests to request more storage (e.g., increase

storage: 1Gi

to a higher value). For example, in the Flyte sandbox, this is controlled by the

charts/flyte-sandbox/templates/storage/minio/pv.yaml

and

pvc.yaml

files, where you can set a larger value for the

capacity.storage

and

resources.requests.storage

fields. After updating, you may need to recreate the PV/PVC or expand the volume if your Kubernetes environment supports dynamic resizing. See pv.yaml and pvc.yaml for reference. Would you like more detailed step-by-step instructions for resizing your MinIO storage in Kubernetes? Sources: - pv.yaml - pvc.yaml 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

I gave 16gi of storage to minio but still it says the same error. IN the minio console, it looks like there are only couple of kb worth of data in there, what could be the issue @ancient-wolf-19325 ?

If you've allocated 16Gi to MinIO but still see the "XMinioStorageFull" error and the MinIO console shows only a few KB of data, it's likely that the underlying disk or host path backing the PersistentVolume is full, or the PV is not correctly mapped to the expected storage location. This can happen if the host directory used for the PV is on a disk with limited space, regardless of the PVC's requested size. Check the actual disk usage on the node where MinIO is running, especially the path specified in your PV's

hostPath

(e.g.,

/var/lib/flyte/storage/minio

). If the host disk is full, MinIO will report storage full even if the PVC requests more space. Also, ensure that the PV and PVC are correctly bound and that MinIO is using the intended volume. See pv.yaml for the hostPath configuration. Would you like more detail on how to check disk usage or debug PV/PVC bindings? Sources: - pv.yaml 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.