@average-finland-92144 Thanks, things line-up pretty well with those scripts. After some further debugging, it seems like simpler task outputs (like python primitives such as lists and dictionaries) are successfully written to s3 bucket, however, only when I'm trying to return a FlyteFile from a task (in this case a pytorch checkpoint stored in the task's local storage) do I get this weird access denied error.
I figured maybe the file was being created with too restrictive rwx permissions, so I changed the file permissions to 777 before trying to return it again, but still got the same error. I then wondered if maybe the issue was how torch.save() was creating the file, so I decided to create a plain-text file and return that instead. To my surprise, this worked, my task completed successfully and I was able to find the file in my s3 storage.
Do you have any idea why this could be happening with torch.save() (maybe something to do with changes to how torch.save works vs what flyte can support?). I've copied
example like this one to return a pytorch checkpoint from a task using torch.save() to create the checkpoint, but always get the permission denied error. Also, let me know if I should move this question to another channel, as this is definitely no longer a deployment issue.