Hi! I’m currently evaluating Flyte and I’m testing some basic use cases. I have a simple use case where: - Task A: Generates some files and a metadata file. - Task B Reads the files produced by Task A and outputs a list of those files. I implemented both tasks using

ContainerTask

. However, when executing the workflow, I notice that the input directory in Task B is empty, even though Task A should have produced files there. However, when I implement the same tasks using the

@task

decorator, everything works as expected. Here is the sample code I’m using:

from flytekit import ContainerTask, workflow, kwtypes
from flytekit.types.directory import FlyteDirectory

# Task A: Writes files to a directory using ContainerTask
task_a = ContainerTask(
    name="task_a",
    image="alpine:latest",
    command=[
        "sh",
        "-c",
        """
        mkdir -p /var/outputs/output_dir
        for i in $(seq 0 4); do
            echo "This is file $i" > /var/outputs/output_dir/file_$i.txt
        done
        """
    ],
    inputs={},
    outputs=kwtypes(output_dir=FlyteDirectory),
    input_data_dir="/var/inputs",
    output_data_dir="/var/outputs",
)

# Task B: Reads files from the directory using ContainerTask
task_b = ContainerTask(
    name="task_b",
    image="alpine:latest",
    command=[
        "sh",
        "-c",
        """
        input_dir=/var/inputs/input_dir
        echo "Input directory: $input_dir"
        files=$(ls $input_dir)
        echo "Files in directory: $files"
        num_files=$(ls -1 $input_dir | wc -l)
        echo "Number of files: $num_files"
        echo $num_files > /var/outputs/result.txt
        """
    ],
    inputs=kwtypes(input_dir=FlyteDirectory),
    outputs=kwtypes(result=int),
    input_data_dir="/var/inputs",
    output_data_dir="/var/outputs",
)

# Workflow that connects Task A and Task B
@workflow
def directory_workflow() -> int:
    dir_output = task_a().output_dir
    result = task_b(input_dir=dir_output).result
    return result

What could be the issue? Thanks!

cc @tall-lock-23197

i am able to reproduce the issue. this should work on flyte 1.14 but it isn't. cc @damp-lion-88352

trying

the reason is that we have a new copilot image, but i think the https://github.com/flyteorg/flyte/blob/7d53a6eb5df1c0e73a5e844b4d90318083ed84ac/flyteplugins/go/tasks/pluginmachinery/flytek8s/config/config.go#L36

I will use this PR's example to debug

Hi!! Thanks for the quick response. When will this feature be released?

following up on this: @damp-lion-88352 have you been able to identify the cause of this issue?

coming