I’m guessing you’ve seen this page? https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/raw_container.html#container-tasks Are you using python in your container task or another language?

So I think the issue is to due with how I’m defining my File inputs and Directory outputs. I’ve got that example you’ve shared working with no issues, however when using a FlyteFile is seems to struggle pulling the file in.

train_task_container = ContainerTask(
    name="regression_container",
    input_data_dir="/var/inputs",
    output_data_dir="/var/outputs",
    inputs=kwtypes(x=FlyteFile),
    outputs=kwtypes(output=FlyteFile),
    image="pachyderm/housing-prices:1.11.0",
    command=[
        "python",
        "regression.py",
        "--input",
        "/var/inputs",
        "--target-col",
        "MEDV",
        "--output",
        "/var/outputs/output",
    ],
)

This is how the step is defined, so its running a python script but failing because no file is picked up. I’ve attatched the copilot downloader output which seems to show this as well. Again this works fine when using python primitive types for inputs and outputs

also, can you share the workflow that uses the

train_task_container

task?

And the workflow:

Untitled.py

I’m basically comparing the ShellTask to the ContainerTask, ShellTask is working fine with the same script

Getting the following error now

Pod failed. No message received from kubernetes.
[flyte-copilot-downloader] terminated with ExitCode 0.
[az6rsxswjlzql79xl2bv-n2-0] terminated with exit code (1). Reason [Error]. Message: 
Traceback (most recent call last):
  File "regression.py", line 101, in <module>
    main()
  File "regression.py", line 79, in main
    print("Datasets: {}".format(input_files))
UnboundLocalError: local variable 'input_files' referenced before assignment
.
[flyte-copilot-sidecar] terminated with ExitCode 0.

And the command executing in container

python regression.py --input <s3://adarga-ds-lab-2-flyte-data/data/2c/a9j9fgb4l89j8h6t9gfr-n0-0/238d789391a50d97f62e7fcf20e9d42c/boston_housing.csv> --target-col MEDV --output /var/outputs/output

So obviously the s3 address is being passed in instead of the file url

To add, for my use case I can use the ShellTask instead of a ContainerTask, but I was just trying to compare them. For us, its a question of which is quicker for migrating old pipelines over to Flyte but theres not much in it, just need to add awscli and flykit deps

i'll continue down this path and let you know if i find a solution

I found a solution. For

ContainerTask

, for a

myinput=FlyteFile

input variable configured for the directory `/var/inputs`:

/var/inputs/myinput

=the file

{{.inputs.myinput}}

=the remote file uri

Your original attempt was

--input /var/inputs

which wasn't right, should have been

--input /var/inputs/x

That works for me when using python primitives, but anything that needs downloading fails. Looking more into this, looks like its the initcontainer that is failing to download from S3, but its returning a 0 exit code which it probably shouldn’t 🤷