Hi all, I’ve been working with Flyte the couple la...
# ask-the-community
m
Hi all, I’ve been working with Flyte the couple last weeks to implement some model workflows. I’m struggling with passing files to a containertask. Is it possible to pass a list of files or directory? I managed to pass 1 FlyteFile, but the model requires multiple files to run. If I pass a List here to the inputs, the copilot-downloader can’t load the files.
Copy code
floodcast_model = ContainerTask(
    name="floodcast-docker",
    input_data_dir="/input",
    output_data_dir="/output",
    inputs=kwtypes(rainfall=List[FlyteFile]),
    outputs=kwtypes(floodmaps=List[FlyteFile]),
    image="<myimage>",
    command=["/env/bin/python", "run.py"])
k
Did copilot show any error?
m
yes, the following error is shown in logs of flyte-copliot-downloader pod.
Copy code
{"json":{},"level":"info","msg":"Downloading inputs from [<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-f6623e1e940a941929ce/n2/data/inputs.pb>]","ts":"2022-09-26T08:52:33Z"}  │
│ {"json":{},"level":"info","msg":"Waiting for [rainfall] to be persisted","ts":"2022-09-26T08:52:33Z"}                                                                                         │
│ {"json":{},"level":"error","msg":"Failed to persist [rainfall], err failed to open file at path /input/rainfall/0: open /input/rainfall/0: no such file or directory","ts":"2022-09-26T08:52: │
│ 33Z"}
k
looks like the path is incorrect, it should be s3 path. Mind sharing you entire code?
m
It does work if I pass 1 FlyteFilte instead of the list. But just noticed an error I overlooked in the copilot-sidecar..
Copy code
{"json":{},"level":"error","msg":"Uploading failed, err currently CoPilot uploader does not support [collection_type:\u003cblob:\u003c\u003e \u003e ], system error","ts":"2022-09-26T08:53:0 │

│ 8Z"}
So it seems this is not supported?
The input files are all FlyteFiles with s3 path.
k
You are right. copilot now only support primitive type and blob type. https://github.com/flyteorg/flytecopilot/blob/69ea09dbf732e3e71a565c8ce448c3168141a454/data/upload.go#L143-L163 mind filing an issue, we’ll support collection type in next release.
m
Ok, thanks for you reply! I’ll create an issue.
k
Thank you
114 Views