Hi, I’m facing an issue related to FlyteDirectory. I have two tasks, task1 basically creates a FlyteDirectory. I can see that data is uploaded in minio. task2 is a ContainerTask which takes the FlyteDirectory created by task1 as input. From the logs of task2 , I can see that it is not able to see the data of FlyteDirectory. I verified the same by logging into the container. Below is the code for the ContainerTask

task2 = ContainerTask(
    name="task2",
    input_data_dir="/var/inputs",
    output_data_dir="/var/outputs",
    inputs=kwtypes(data=FlyteDirectory),
    outputs=kwtypes(out=FlyteDirectory),
    image="task2-image....",
    command=[...],
    arguments=[...]
)

@task
def task1() -> FlyteDirectory:
 ...
@workflow
def aeb_sanity_workflow() -> FlyteDirectory:
    data = task1()
    ans = task2(data=data)

Just to check the functionality of FlyteDirectory, I tried creating another task3 which is normal python task which can take this FlyteDirectory as input, and iterate over that using

os.listdir

. I’m able to see the files in this task. What exactly I need to do in ContainerTask so that FlyteDirectory content could be seen at

/var/inputs/data

i think this is a limitation of copilot that we need to address.

I’ve found a workaround. I’m using task3 to fetchData from FlyteDirectory and then saving that data in host machine (docker container in this case, since i’m using flyte sandbox installation) and then it turn sharing this again with ContainerTask using volume mount

Also, please check out https://gist.github.com/pryce-turner/0a67f86febdc812c9a2a9e739c22eeca written by one of our community members. Isn't exactly the same as what you're trying to do but can be helpful.

Yes, I read that. It was really helpful.

Hello #Gaurav Kumar, were you able to run copilot with minio S3 or did you use AWS s3