https://flyte.org logo
#ask-the-community
Title
# ask-the-community
c

Cornelis Boon

03/18/2024, 4:04 PM
Uh. Is it true that I can't use
FlyteDirectory
type inputs for `ContainerTask`s?
When I try to do this, I get:
Copy code
Currently only single part blobs are supported, we will force multipart to be 'path/00000'"
"Failed to download from ref [<gs://flyte-bucket/data/r5/apv2mgflkqlq9pp6rz2t-n1-0-n1-n0-0-n0-0/90aa7e15e7ec2c26f237238d38632a4a/000000>]"
"Failed to persist [input_dir], err incorrect blob reference, does not exist", "ts":"2024-03-18T16:01:45Z"
"Downloading failed, err failed to download input variable from remote store: variable [input_dir] download/store failed: incorrect blob reference, does not exist"
And I'm guessing `FlyteDirectory`s are also not allowed as outputs of `ContainerTask`s
k

Kevin Su

03/18/2024, 7:03 PM
cc @L godlike we should support FlyteFile/FlyteDirectory in local execution as well
c

Cornelis Boon

03/18/2024, 10:30 PM
FlyteDirectory works locally, afaik
At least did not have issues with local FlyteDirectory inputs/outputs with python tasks. But I guess that's also true for remote
k

Kevin Su

03/18/2024, 10:31 PM
yes, but we can’t run
ContainerTask
locally for now
c

Cornelis Boon

03/18/2024, 10:31 PM
Ah, yeah. Would be great to test such workflows e2e locally too
k

Kevin Su

03/18/2024, 10:32 PM
c

Cornelis Boon

03/18/2024, 10:38 PM
For now I'm thinking to make a workaround that where either the task preceding the container task outputs a
List[FlyteFile]
and then just map a task over this list and recombine the outputs. Still, being able to use
FlyteDirectory
directly and letting this mapping happen in the container (as the number of Files to be processed is small) would be more convenient
l

L godlike

03/19/2024, 12:16 AM
I think
List
and
Dict
is not supported in Container task even the code
flytecopilot
seems implemented them well
This PR can help you test
FlyteFile
and
FlyteDirectory
's local execution
c

Cornelis Boon

03/19/2024, 10:38 AM
I think
List
and
Dict
is not supported in Container task
even the code
flytecopilot
seems implemented them well
But individual `FlyteFile`s should be fine, right? like when a
List
of
FlyteFile
s is mapped using something like
map_task
or
array_node_map_task
right?
l

L godlike

03/19/2024, 12:57 PM
Yes
For map task, there is a PR supporting container task with map task
c

Cornelis Boon

03/19/2024, 12:58 PM
Wait. So that's also not supported yet? Hmm.
l

L godlike

03/19/2024, 12:59 PM
Yes I think we should support it
container task needs to implement lots of complex logic
so it will be limited support
c

Cornelis Boon

03/19/2024, 3:28 PM
Eventually circumvented my issue by creating an imagespec with the container as the base image and then creating a PythonTask that uses that imagespec to call the relevant applications in the image and handle inputs/outputs
l

L godlike

03/27/2024, 12:42 AM
It makes sense, it's doable
3 Views