hey all! quick storage question, does flyte provid...
# ask-the-community
m
hey all! quick storage question, does flyte provide any way to automatically delete a file from blob storage once it’s been successfully read in from a new task? I pretty much have a few tasks that transform data in a file, and my first preference would be to avoid saving down the intermediate data altogether, as it’s quite sensitive. my understanding is that it needs to be saved down somewhere outside the task pod, so the next best thing would be to delete it after it’s read in by the subsequent task
k
You can simply add s3 lifecycle policies, keep it around for like a week and delete?
I would use a new raw bucket and just set the policy
m
thanks! I think this is what we’ll end up doing.. we’re on GCS and I believe the minimum retention is 1 day, it’s possible we’ll need the files to be deleted within a shorter timeframe but if that’s the case I think we’ll write a cleanup flyte task that deletes any intermediate files and add that to the end of the graph
k
Yup that can be done
You can also listen to events and delete all intermediaries
105 Views