Hi all, I’ve been working with Flyte the couple last weeks to implement some model workflows. I’m struggling with passing files to a containertask. Is it possible to pass a list of files or directory? I managed to pass 1 FlyteFile, but the model requires multiple files to run. If I pass a List here to the inputs, the copilot-downloader can’t load the files.

floodcast_model = ContainerTask(
    name="floodcast-docker",
    input_data_dir="/input",
    output_data_dir="/output",
    inputs=kwtypes(rainfall=List[FlyteFile]),
    outputs=kwtypes(floodmaps=List[FlyteFile]),
    image="<myimage>",
    command=["/env/bin/python", "run.py"])

Did copilot show any error?

yes, the following error is shown in logs of flyte-copliot-downloader pod.

{"json":{},"level":"info","msg":"Downloading inputs from [<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-f6623e1e940a941929ce/n2/data/inputs.pb>]","ts":"2022-09-26T08:52:33Z"}  │
│ {"json":{},"level":"info","msg":"Waiting for [rainfall] to be persisted","ts":"2022-09-26T08:52:33Z"}                                                                                         │
│ {"json":{},"level":"error","msg":"Failed to persist [rainfall], err failed to open file at path /input/rainfall/0: open /input/rainfall/0: no such file or directory","ts":"2022-09-26T08:52: │
│ 33Z"}

looks like the path is incorrect, it should be s3 path. Mind sharing you entire code?

It does work if I pass 1 FlyteFilte instead of the list. But just noticed an error I overlooked in the copilot-sidecar..

You are right. copilot now only support primitive type and blob type. https://github.com/flyteorg/flytecopilot/blob/69ea09dbf732e3e71a565c8ce448c3168141a454/data/upload.go#L143-L163 mind filing an issue, we’ll support collection type in next release.

Ok, thanks for you reply! I’ll create an issue.

Thank you