(Update) We now support input/output interpolation in

SlurmShellTask

! Here’s an example task:

write_file_task = SlurmShellTask(
    name="write-file",
    script="""#!/bin/bash

echo "[SlurmShellTask] Write something into a file..." >> {inputs.x}
if grep "file" {inputs.x}
then
    echo "Found a string 'file'!" >> {inputs.x}
else
    echo "'file' not found!"
fi
    """,
    task_config=Slurm(
        ssh_config={
            "host": "aws2",
            "username": "ubuntu",
        },
        sbatch_conf={
            "partition": "debug",
            "job-name": "tiny-slurm",
        }
    ),
    inputs=kwtypes(x=str),
    output_locs=[OutputLocation(var="i", var_type=FlyteFile, location="{inputs.x}")],
)

A workflow that demonstrates passing files between tasks: https://github.com/JiangJiaWei1103/Flyte-Demos/blob/main/slurm_agent/script/shell_3.py Observations when passing files between tasks: 1. FlyteFile as an input type is problematic – The file will be created on the Slurm cluster and can't be found on the local machine 2. Output location type mismatch – Even though

var_type

in

output_locs

is set to

FlyteFile

, it’s actually a

str

I’d love to hear your thoughts on this. Thanks!

great work When do you think this integration would be somewhat ready for public promotion? Like writing a blog and/or giving a presentation to the community. We can help you either way Thanks!

[after sync with abao] you should follow the behavior here for slurm task https://docs.flyte.org/en/latest/user_guide/basics/shell_tasks.html#shell-task

Hi David, I appreciate the support. I’d love to share this with the community, but I think it still needs some polishing (e.g., bug fix, better error handling, etc.). Maybe after the first version is merged, we can start planning a presentation—what do you think?

@creamy-shampoo-53278 that sounds great, thank you!

I'm getting back into this, it's coming along nicely! Is it still good to work off of

slurm-agent-dev

branch? Seems most active. Also, are the scripts in the flyte-demos repo above the most up to date examples moving forward?

I;m going to merge slurm-funciton task today https://github.com/flyteorg/flytekit/pull/3150#pullrequestreview-2634673579