https://flyte.org
Join Slack
Uh. Is it true that I can't use `FlyteDirectory` type inputs for `ContainerTask`s?
# flyte-support
g
Uh. Is it true that I can't use 
FlyteDirectory
type inputs for `ContainerTask`s?
When I try to do this, I get:
Copy code
Currently only single part blobs are supported, we will force multipart to be 'path/00000'"
"Failed to download from ref [<gs://flyte-bucket/data/r5/apv2mgflkqlq9pp6rz2t-n1-0-n1-n0-0-n0-0/90aa7e15e7ec2c26f237238d38632a4a/000000>]"
"Failed to persist [input_dir], err incorrect blob reference, does not exist", "ts":"2024-03-18T16:01:45Z"
"Downloading failed, err failed to download input variable from remote store: variable [input_dir] download/store failed: incorrect blob reference, does not exist"
And I'm guessing `FlyteDirectory`s are also not allowed as outputs of `ContainerTask`s
g
cc @damp-lion-88352 we should support FlyteFile/FlyteDirectory in local execution as well
g
FlyteDirectory works locally, afaik
At least did not have issues with local FlyteDirectory inputs/outputs with python tasks. But I guess that's also true for remote
g
yes, but we can’t run 
ContainerTask
locally for now
g
Ah, yeah. Would be great to test such workflows e2e locally too
g
@damp-lion-88352 is working on it. https://github.com/flyteorg/flytekit/pull/2258
g
For now I'm thinking to make a workaround that where either the task preceding the container task outputs a 
List[FlyteFile]
and then just map a task over this list and recombine the outputs. Still, being able to use 
FlyteDirectory
directly and letting this mapping happen in the container (as the number of Files to be processed is small) would be more convenient
d
I think 
List
and 
Dict
is not supported in Container task even the code 
flytecopilot
seems implemented them well
This PR can help you test 
FlyteFile
and 
FlyteDirectory
's local execution
g
I think 
List
and 
Dict
is not supported in Container task
even the code 
flytecopilot
seems implemented them well
But individual `FlyteFile`s should be fine, right? like when a 
List
of 
FlyteFile
s is mapped using something like 
map_task
or 
array_node_map_task
right?
d
Yes
For map task, there is a PR supporting container task with map task
g
Wait. So that's also not supported yet? Hmm.
d
Yes I think we should support it
container task needs to implement lots of complex logic
so it will be limited support
g
Eventually circumvented my issue by creating an imagespec with the container as the base image and then creating a PythonTask that uses that imagespec to call the relevant applications in the image and handle inputs/outputs
d
It makes sense, it's doable
53 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by