(Update)
We now support input/output interpolation in
SlurmShellTask
! Here’s an example task:
write_file_task = SlurmShellTask(
name="write-file",
script="""#!/bin/bash
echo "[SlurmShellTask] Write something into a file..." >> {inputs.x}
if grep "file" {inputs.x}
then
echo "Found a string 'file'!" >> {inputs.x}
else
echo "'file' not found!"
fi
""",
task_config=Slurm(
ssh_config={
"host": "aws2",
"username": "ubuntu",
},
sbatch_conf={
"partition": "debug",
"job-name": "tiny-slurm",
}
),
inputs=kwtypes(x=str),
output_locs=[OutputLocation(var="i", var_type=FlyteFile, location="{inputs.x}")],
)
A workflow that demonstrates passing files between tasks:
https://github.com/JiangJiaWei1103/Flyte-Demos/blob/main/slurm_agent/script/shell_3.py
Observations when passing files between tasks:
1.
FlyteFile as an input type is problematic – The file will be created on the Slurm cluster and can't be found on the local machine
2.
Output location type mismatch – Even though
var_type
in
output_locs
is set to
FlyteFile
, it’s actually a
str
I’d love to hear your thoughts on this. Thanks!
