@ancient-wolf-19325 Shelltasks return Tuple[FlyteFile, FlyteFile]. How can I get the file paths as strings. I get errors about transformers

To get the file paths as strings from a

ShellTask

that returns

Tuple[FlyteFile, FlyteFile]

, you need to handle the

FlyteFile

objects correctly. The issue you're encountering is likely due to the way Flyte handles type transformations. You can use the

path

attribute of the

FlyteFile

objects to get the file paths as strings. Here's an example of how you can achieve this:

from flytekit import kwtypes, task, dynamic
from flytekit.extras.tasks.shell import OutputLocation, ShellTask
from flytekit.types.file import FlyteFile

# Define the ShellTask
st = ShellTask(
    name="shell_task",
    debug=True,
    script="""
    touch {inputs.filename}
    """,
    inputs=kwtypes(filename=str),
    output_locs=[OutputLocation(var="outfile", var_type=FlyteFile, location="{inputs.filename}")],
)

# Define a dynamic task to handle the ShellTask output
@dynamic
def t1_task() -> str:
    outfile = st(filename='test.txt')
    return outfile.path

# Run the dynamic task
if __name__ == "__main__":
    print(f"Running t1_task() {t1_task()}")

In this example, the

ShellTask

creates a file, and the dynamic task

t1_task

returns the file path as a string using the

path

attribute of the

FlyteFile

object. Would you like more details on this or any other aspect of Flyte? Sources: - BUG: ShellTasks return promises when called inside a task 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 The outputlocs is a Tuple

output_locs=[
        OutputLocation(var="stdout", var_type=FlyteFile, location="stdout.txt"),
        OutputLocation(var="stderr", var_type=FlyteFile, location="stderr.txt"),
    ],

To handle a

ShellTask

that returns a tuple of

FlyteFile

objects, you can access each file's path using the

path

attribute. Here's an example of how to handle multiple

FlyteFile

outputs:

from flytekit import kwtypes, task, dynamic
from flytekit.extras.tasks.shell import OutputLocation, ShellTask
from flytekit.types.file import FlyteFile

# Define the ShellTask with multiple output locations
st = ShellTask(
    name="shell_task",
    debug=True,
    script="""
    echo "This is stdout" > stdout.txt
    echo "This is stderr" > stderr.txt
    """,
    inputs=kwtypes(),
    output_locs=[
        OutputLocation(var="stdout", var_type=FlyteFile, location="stdout.txt"),
        OutputLocation(var="stderr", var_type=FlyteFile, location="stderr.txt"),
    ],
)

# Define a dynamic task to handle the ShellTask output
@dynamic
def t1_task() -> tuple[str, str]:
    stdout, stderr = st()
    return stdout.path, stderr.path

# Run the dynamic task
if __name__ == "__main__":
    stdout_path, stderr_path = t1_task()
    print(f"stdout path: {stdout_path}")
    print(f"stderr path: {stderr_path}")

In this example, the

ShellTask

generates two files:

stdout.txt

and

stderr.txt

. The dynamic task

t1_task

captures these outputs and returns their paths as strings. Would you like more details on this or any other aspect of Flyte? Sources: - BUG: ShellTasks return promises when called inside a task 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 the code you supplied returns

RestrictedTypeError: Transformer for type <class 'tuple'> is restricted currently

The error you're encountering is due to Flyte's current restriction on handling tuples directly. To work around this, you can use a

dataclass

to encapsulate the multiple outputs instead of returning a tuple directly. Here's how you can modify the code to use a `dataclass`:

from dataclasses import dataclass
from flytekit import kwtypes, task, dynamic
from flytekit.extras.tasks.shell import OutputLocation, ShellTask
from flytekit.types.file import FlyteFile

# Define a dataclass to encapsulate the outputs
@dataclass
class ShellTaskOutputs:
    stdout: FlyteFile
    stderr: FlyteFile

# Define the ShellTask with multiple output locations
st = ShellTask(
    name="shell_task",
    debug=True,
    script="""
    echo "This is stdout" > stdout.txt
    echo "This is stderr" > stderr.txt
    """,
    inputs=kwtypes(),
    output_locs=[
        OutputLocation(var="stdout", var_type=FlyteFile, location="stdout.txt"),
        OutputLocation(var="stderr", var_type=FlyteFile, location="stderr.txt"),
    ],
    output=ShellTaskOutputs
)

# Define a dynamic task to handle the ShellTask output
@dynamic
def t1_task() -> tuple[str, str]:
    outputs = st()
    return outputs.stdout.path, outputs.stderr.path

# Run the dynamic task
if __name__ == "__main__":
    stdout_path, stderr_path = t1_task()
    print(f"stdout path: {stdout_path}")
    print(f"stderr path: {stderr_path}")

In this example, the

ShellTask

is modified to return a

ShellTaskOutputs

dataclass, which encapsulates the

stdout

and

stderr

FlyteFile

objects. The dynamic task

t1_task

then accesses the paths of these files. Would you like more details on this or any other aspect of Flyte? Sources: - Is this expected or are there workarounds? - Quick question about the type system 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

I needed

--overwrite-cache