hey all! quick storage question, does flyte provide any way to automatically delete a file from blob storage once it’s been successfully read in from a new task? I pretty much have a few tasks that transform data in a file, and my first preference would be to avoid saving down the intermediate data altogether, as it’s quite sensitive. my understanding is that it needs to be saved down somewhere outside the task pod, so the next best thing would be to delete it after it’s read in by the subsequent task

You can simply add s3 lifecycle policies, keep it around for like a week and delete?

thanks! I think this is what we’ll end up doing.. we’re on GCS and I believe the minimum retention is 1 day, it’s possible we’ll need the files to be deleted within a shorter timeframe but if that’s the case I think we’ll write a cleanup flyte task that deletes any intermediate files and add that to the end of the graph

Yup that can be done