acoustic-carpenter-78188
12/05/2023, 1:37 AMpyflyte run
with a workflow that uses FlyteFile
or FlyteDirectory
, the type engine will kick in and automatically create temporary copies of the file/directory. This adds an extra layer of indirection, esp. when a user is prototyping locally and explicitly specifying a local path. A recent use case for this is if a user wants to do continued ML training on a single directory that adds model state without deleting prior progress @thomasjpfan.
Goal: What should the final outcome look like, ideally?
When running tasks/workflows locally (e.g. with pyflyte run), the task/workflow should just use the local file/directory without the added step of creating a temporary filepath.
Describe alternatives you've considered
Keeping the status quo will persist the problem mentioned in the motivation.
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteacoustic-carpenter-78188
12/05/2023, 1:37 AM