Yubo Wang

05/18/2023, 6:47 PM
Can anyone in the community provide some rough metrics on the Blob Storage usage by Flyte? For example: total # of workflows, executions, tasks is raw user data stored in the same blob storage? and the existing amount of storage used. Also will appreciate it a lot if someone can share experiences where the blob storage becomes the bottleneck/where failure happens.

Ketan (kumare3)

05/19/2023, 4:23 AM
@Yubo Wang blob storage usage can grow quite a bit, but caching, passing references reduces usage quite a bit. Most cloud blob stores have a expiration policies, which works great with Flyte for completed workflows are data is never reused that way - you can simply delete data after a period