gray-ocean-62145
03/06/2023, 9:53 PMbroad-monitor-993
03/06/2023, 11:54 PMbroad-monitor-993
03/06/2023, 11:55 PMgray-ocean-62145
03/07/2023, 9:32 AMgray-ocean-62145
03/07/2023, 9:33 AMtrain_task_container = ContainerTask(
name="regression_container",
input_data_dir="/var/inputs",
output_data_dir="/var/outputs",
inputs=kwtypes(x=FlyteFile),
outputs=kwtypes(output=FlyteFile),
image="pachyderm/housing-prices:1.11.0",
command=[
"python",
"regression.py",
"--input",
"/var/inputs",
"--target-col",
"MEDV",
"--output",
"/var/outputs/output",
],
)
This is how the step is defined, so its running a python script but failing because no file is picked up. I’ve attatched the copilot downloader output which seems to show this as well. Again this works fine when using python primitive types for inputs and outputsbroad-monitor-993
03/07/2023, 1:39 PMbroad-monitor-993
03/07/2023, 3:07 PMtrain_task_container
task?gray-ocean-62145
03/07/2023, 3:13 PM/var/inputs/
is emptygray-ocean-62145
03/07/2023, 3:13 PMgray-ocean-62145
03/07/2023, 3:13 PMgray-ocean-62145
03/07/2023, 3:14 PMbroad-monitor-993
03/07/2023, 7:52 PM--input /var/inputs
might be the issue here: can you use the templating syntax --inputs {{.inputs.x}}
in train_task_container
instead?
Basically the current code points to the /var/inputs
directory whereas {{.inputs.x}}
will inject the correct filepathgray-ocean-62145
03/07/2023, 8:16 PMgray-ocean-62145
03/07/2023, 8:17 PMPod failed. No message received from kubernetes.
[flyte-copilot-downloader] terminated with ExitCode 0.
[az6rsxswjlzql79xl2bv-n2-0] terminated with exit code (1). Reason [Error]. Message:
Traceback (most recent call last):
File "regression.py", line 101, in <module>
main()
File "regression.py", line 79, in main
print("Datasets: {}".format(input_files))
UnboundLocalError: local variable 'input_files' referenced before assignment
.
[flyte-copilot-sidecar] terminated with ExitCode 0.
And the command executing in container
python regression.py --input <s3://adarga-ds-lab-2-flyte-data/data/2c/a9j9fgb4l89j8h6t9gfr-n0-0/238d789391a50d97f62e7fcf20e9d42c/boston_housing.csv> --target-col MEDV --output /var/outputs/output
So obviously the s3 address is being passed in instead of the file urlabundant-laptop-64153
03/17/2023, 5:13 PMgray-ocean-62145
03/17/2023, 5:51 PMgray-ocean-62145
03/17/2023, 5:53 PMabundant-laptop-64153
03/17/2023, 5:54 PMabundant-laptop-64153
03/17/2023, 5:54 PMabundant-laptop-64153
03/17/2023, 6:07 PMContainerTask
, for a myinput=FlyteFile
input variable configured for the directory `/var/inputs`:
/var/inputs/myinput
=the file
{{.inputs.myinput}}
=the remote file uriabundant-laptop-64153
03/17/2023, 6:07 PM--input /var/inputs
which wasn't right, should have been --input /var/inputs/x
gray-ocean-62145
03/20/2023, 8:15 AMgray-ocean-62145
03/20/2023, 11:01 AMcolossal-cricket-16762
04/15/2023, 8:05 PM/tmp/inputs
is mounted into input_data_dir="/var/inputs"
of the docker container, but I can't figure out how to download the file into /tmp/inputs
with a ContainerTask.abundant-laptop-64153
04/15/2023, 8:22 PM/var/inputs/myvar
colossal-cricket-16762
04/15/2023, 8:48 PM