https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
e

Eli Bixby

01/26/2023, 11:18 AM
Has anyone done 
FlyteDirectory
inputs to a 
ContainerTask
? It seems like the download step doesn't correctly handle directories? with 
inputs=dict(path=FlyteDirectory)
we're getting 
Failed to download from ref [<gs://cradle-bio-pipelines/i9/f7aac4450113a425b905-n0-0/path>]
where that path does exist.
Following up on this. Problem was that the task didn't have an output declared, so the sidecar container exited immediately without downloading the input.
k

Ketan (kumare3)

01/26/2023, 3:16 PM
Ohh this is a bug, we are tracking to fix this month
f

Felix Ruess

01/26/2023, 3:27 PM
but the input is downloaded by the init container right? the sidecar is only for uploading outputs...?
@Eli Bixby see https://github.com/flyteorg/flyte/issues/3254 for the early sidecar exit... But I think this is not the cause of your problem, rather the sidecar shouldn't even be started if you don't have any output.
d

Dan Rammer (hamersaw)

01/26/2023, 3:40 PM
@Eli Bixby and @Felix Ruess can you provide examples of the ContainerTask definition where these issues are occurring?
k

Ketan (kumare3)

01/26/2023, 3:40 PM
@Dan Rammer (hamersaw) I know
It’s a one line fix in the way we create the copilot job
I can create the PR or you can today
d

Dan Rammer (hamersaw)

01/26/2023, 3:42 PM
I have this on my TODO list for today, just easier to get test cases rather than write my own 😛
f

Felix Ruess

01/26/2023, 3:43 PM
added a simple container tasks without outputs to the issue
But it seems @Eli Bixby has a different issue...
d

Dan Rammer (hamersaw)

01/26/2023, 3:44 PM
yeah, i agree. i'll take try to get them both wrapped up in the fix.
@Eli Bixby / @Felix Ruess what version of k8s are you running? In testing here I'm seeing some interesting behavior with different versions that need to be sorted out.
f

Felix Ruess

01/26/2023, 6:21 PM
1.24 or to be specific v1.24.6+k3s
e

Eli Bixby

01/30/2023, 1:19 PM
Same
p

Pryce

04/24/2023, 12:09 AM
Hey @Eli Bixby, wondering if you could share your code for how you got this working with a FlyteDirectory input? I've tried it every way I can think of and nothing shows up in the container itself...
Copy code
bt = ContainerTask(
    name="basic-test",
    input_data_dir="/var/inputs",
    output_data_dir="/var/outputs",
    inputs=kwtypes(indir=FlyteDirectory),
    outputs=kwtypes(),
    image="<http://ghcr.io/flyteorg/rawcontainers-shell:v2|ghcr.io/flyteorg/rawcontainers-shell:v2>",
    command=[
        "ls",
        "-la",
        "/var/inputs",
    ],
)

@task
def get_dir(dirpath: str) -> FlyteDirectory:
    fd = FlyteDirectory(path=dirpath)
    return fd

@workflow
def wf():
    fd = get_dir(dirpath='<s3://my-s3-bucket/cv-in>')
    bt(indir=fd)
I also tried specifying outputs per your original follow-up, but that didn't help. I appreciate any insight you can provide!
e

Eli Bixby

04/27/2023, 8:45 AM
Hey @Pryce, I think we decided to go a different way with the workflow we needed this for. I'll double check with my team, but I'm fairly sure we never got this working.
p

Pryce

04/29/2023, 4:03 AM
Ah shame - oh well, thanks for following up!
Powered by