Michael Tinsley
03/06/2023, 9:53 PMNiels Bantilan
03/06/2023, 11:54 PMMichael Tinsley
03/07/2023, 9:32 AMtrain_task_container = ContainerTask(
name="regression_container",
input_data_dir="/var/inputs",
output_data_dir="/var/outputs",
inputs=kwtypes(x=FlyteFile),
outputs=kwtypes(output=FlyteFile),
image="pachyderm/housing-prices:1.11.0",
command=[
"python",
"regression.py",
"--input",
"/var/inputs",
"--target-col",
"MEDV",
"--output",
"/var/outputs/output",
],
)
This is how the step is defined, so its running a python script but failing because no file is picked up. I’ve attatched the copilot downloader output which seems to show this as well. Again this works fine when using python primitive types for inputs and outputsNiels Bantilan
03/07/2023, 1:39 PMtrain_task_container
task?Michael Tinsley
03/07/2023, 3:13 PM/var/inputs/
is emptyNiels Bantilan
03/07/2023, 7:52 PM--input /var/inputs
might be the issue here: can you use the templating syntax --inputs {{.inputs.x}}
in train_task_container
instead?
Basically the current code points to the /var/inputs
directory whereas {{.inputs.x}}
will inject the correct filepathMichael Tinsley
03/07/2023, 8:16 PMPod failed. No message received from kubernetes.
[flyte-copilot-downloader] terminated with ExitCode 0.
[az6rsxswjlzql79xl2bv-n2-0] terminated with exit code (1). Reason [Error]. Message:
Traceback (most recent call last):
File "regression.py", line 101, in <module>
main()
File "regression.py", line 79, in main
print("Datasets: {}".format(input_files))
UnboundLocalError: local variable 'input_files' referenced before assignment
.
[flyte-copilot-sidecar] terminated with ExitCode 0.
And the command executing in container
python regression.py --input <s3://adarga-ds-lab-2-flyte-data/data/2c/a9j9fgb4l89j8h6t9gfr-n0-0/238d789391a50d97f62e7fcf20e9d42c/boston_housing.csv> --target-col MEDV --output /var/outputs/output
So obviously the s3 address is being passed in instead of the file urlJustin Boutwell
03/17/2023, 5:13 PMMichael Tinsley
03/17/2023, 5:51 PMJustin Boutwell
03/17/2023, 5:54 PMContainerTask
, for a myinput=FlyteFile
input variable configured for the directory `/var/inputs`:
/var/inputs/myinput
=the file
{{.inputs.myinput}}
=the remote file uri--input /var/inputs
which wasn't right, should have been --input /var/inputs/x
Michael Tinsley
03/20/2023, 8:15 AMLi Tai Fang
04/15/2023, 8:05 PM/tmp/inputs
is mounted into input_data_dir="/var/inputs"
of the docker container, but I can't figure out how to download the file into /tmp/inputs
with a ContainerTask.Justin Boutwell
04/15/2023, 8:22 PM/var/inputs/myvar
Li Tai Fang
04/15/2023, 8:48 PM