https://flyte.org logo
Join Slack
Powered by
How can you override task resources to a shellTask...
# flyte-support
b
How can you override task resources to a shellTask? If I call a shelltask from another task I get a warning on the pod about nested tasks I think it might be task_config but can't find documentation
Copy code
shell_task = ShellTask(
    name="shell_task",
    debug=True,
    script="""
    {inputs.command} >> {outputs.stdout} 2>> {outputs.stderr}
    """,
    inputs=kwtypes(command=str),
    output_locs=[
        OutputLocation(var="stdout", var_type=FlyteFile, location="stdout.txt"),
        OutputLocation(var="stderr", var_type=FlyteFile, location="stderr.txt"),
    ],
)

@task(
    requests=Resources(cpu="4", mem="16Gi")
)
def call_shell_task(....
"name": "flytekit", "levelname": "ERROR", "message": "You are not supposed to nest @Task/@Workflow inside a @Task!"}
t
in general, you shouldn’t call a task from within another task. this is not supported behavior.
the only time this is possible is in the dynamic case (because it becomes a new workflow) and in the eager case (wherein the parent task becomes its own orchestrator).
b
how would I define resources for a shell task ? I think it might be possible with pod templates
t
ShellTask should take requests/limits just like a function task.
along with pod template and pod template name
b
I'm guessing thats by using task_config then. Its not well documented
t
no these should be just args to the ShellTask constructor, just like inputs… they’re hidden by 
kwargs
so you have to follow up the stack. And yeah, docs could use a boost
b
for anyone that makes it here via a search the solution is
Copy code
shell_task = ShellTask(
    name="shell_task",
    debug=True,
    script="""
    {inputs.command} >> {outputs.stdout} 2>> {outputs.stderr}
    """,
    inputs=kwtypes(command=str),
    output_locs=[
        OutputLocation(var="stdout", var_type=FlyteFile, location="stdout.txt"),
        OutputLocation(var="stderr", var_type=FlyteFile, location="stderr.txt"),
    ],
    requests=Resources(cpu="2", mem="16Gi")
)
I have tried to use map_tasks with ShellTask. I don't think its supported, the error I got was the output, ShellTasks require two default outputs, stdout and stderr.
Copy code
ValueError: Error encountered while executing 'load':
  Only tasks with a single output are supported in map tasks.
Are Shell tasks just running python subprocess? https://github.com/flyteorg/flytekit/blob/2dcbb90f7a331eb2d75a826053fc41f944ad4aa2/flytekit/extras/tasks/shell.py#L52 I can use a regular task that runs python subprocess instead of ShellTask and map that task. What advantages do ShellTasks have ?
7 Views
Open in Slack
PreviousNext
Powered by