@ancient-wolf-19325 In my task I am creating a folder to write results into it. This works fine locally, but when running it with

--remote

I receive a PermissionError:

File "/opt/micromamba/envs/runtime/lib/python3.11/pathlib.py", line 1116, in mkdir
        os.mkdir(self, mode)
    PermissionError: [Errno 13] Permission denied: '/Users'

The Path is given via a

Path

from

Pathlib

and should point to the root-directory of the repository. Should this be done differently?

It seems likely that the issue you're encountering is related to permission settings when trying to create a directory in a location that is not writable in the remote environment. When running tasks remotely, the environment may not have the same directory structure or permissions as your local setup. My understanding is that you should consider using a directory path that is guaranteed to be writable in the remote environment, such as a temporary directory or a specific path provided by Flyte for storing outputs. You can use

FlyteDirectory

to handle directory paths in a way that is compatible with both local and remote executions. Would you like more details on how to implement this change or any other aspect of Flyte's directory handling? Sources: - https://github.com/flyteorg/flyte/issues/5426 - https://github.com/flyteorg/flyte/issues/4414 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 I still get a PermissionError when trying to create a folder:

File "/opt/micromamba/envs/runtime/lib/python3.11/pathlib.py", line 1116, in mkdir
        os.mkdir(self, mode)
    PermissionError: [Errno 13] Permission denied: '/var/folders'

It seems likely that the issue is due to attempting to create a directory in a location that is not writable in the remote environment. In remote environments, certain directories may have restricted permissions, preventing the creation of new directories. My understanding is that you should use a directory path that is writable in the remote environment. Consider using a temporary directory or a path provided by Flyte for storing outputs. The

FlyteDirectory

type can be used to manage directory paths in a way that is compatible with both local and remote executions. Would you like more details on how to implement this change or any other aspect of Flyte's directory handling? Sources: - https://github.com/flyteorg/flyte/issues/2771 - https://github.com/flyteorg/flyte/issues/2097 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 This is the code that produces a PermissionError:

@project_task()
def construct_output_directory(root_dir: Path, feature: str, execution_date: str) -> Path:
    return root_dir / "data" / feature / execution_date

@workflow()
def pipeline(feature: str) -> None:
    output_directory = construct_output_directory(
        root_dir=Path(fl.current_context().working_directory),
        feature=feature,
        execution_date=fl.current_context().execution_date.strftime("%Y-%m-%d--%H-%M-%S"),
    )

It seems likely that the issue arises from using

fl.current_context().working_directory

as the root directory, which may not be writable in the remote environment. The

working_directory

in Flyte's context might not always point to a location where you have write permissions, especially in a remote setup. My understanding is that you should consider using a directory path that is guaranteed to be writable, such as a temporary directory provided by Flyte. You can use the

FlyteDirectory

type to manage directory paths in a way that is compatible with both local and remote executions. Would you like more details on how to implement this change or any other aspect of Flyte's directory handling? Sources: - https://github.com/flyteorg/flyte/issues/2097 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 Please give me an example of how to create a writable directory

To create a writable directory in Flyte, you can use the

FlyteDirectory

type, which is designed to handle directory paths in a way that is compatible with both local and remote executions. Here's an example of how you can create a writable directory using `FlyteDirectory`:

from flytekit import task, workflow
from flytekit.types.directory import FlyteDirectory
import os

@task
def create_writable_directory() -> FlyteDirectory:
    # Use Flyte's context to get a writable directory path
    working_dir = flytekit.current_context().working_directory
    writable_dir = os.path.join(working_dir, "my_writable_directory")
    os.makedirs(writable_dir, exist_ok=True)
    return FlyteDirectory(path=writable_dir)

@workflow
def my_workflow() -> FlyteDirectory:
    return create_writable_directory()

This example demonstrates how to create a directory within the Flyte working directory, ensuring that it is writable in both local and remote environments. Would you like more details on this or any other aspect of Flyte's directory handling? Sources: - FlyteDirectory Example 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 Creating the folder in the task works, but using

flytekit.current_context().working_directory

and handing it over to a task results in the permissions error. Is that a bug?

Using

flytekit.current_context().working_directory

directly in a workflow and passing it to a task can lead to permission issues, especially in remote environments. This is because the working directory in the context of a workflow might not be the same or writable in the task's execution environment. My understanding is that you should create the directory within the task itself, where the execution context is guaranteed to have the correct permissions. This approach ensures that the directory is created in a location that is writable by the task. Would you like more details on how to implement this change or any other aspect of Flyte's directory handling? Sources: - https://github.com/flyteorg/flyte/issues/2097 - https://github.com/flyteorg/flyte/issues/6393 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.