https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
s

Sam Eckert

01/30/2023, 11:21 PM
I'm trying to understand the difference between task and workflow at a fundamental level. My understanding was that tasks could not call other tasks. I wrote the following dummy example:
Copy code
@task
def test(x: int) -> int:
    return x + 1

@task
def echo_integer(value: int) -> int:
    logging.error(f"I've got value {value}")
    return test(x=value + 5) + 1

@workflow
def wf(
    value: int,
):
    return echo_integer(value=value)
When run locally, we see type-errors because the 
echo_integer
task now returns a promise instead of an int. Interestingly, it works fine when we run remotely, with the caveat that the called task is not run as a isolated container. I would have expected it to fail remotely as well. Any pointers about what I'm missing in my understanding?
d

Dan Rammer (hamersaw)

01/30/2023, 11:33 PM
so you're exactly correct. 
task is not run as a isolated container.
so running this how you described means that in remote, when the internal task is called, it's called as a python function rather than the starting a new Pod for a Flyte task. So you could essentially remove the 
@task
decorator for the same functionality. So the inputs / outputs will not be serialized, that task will not be registered / versioned, the task status' will not be individually viewable, etc.
n

Niels Bantilan

01/30/2023, 11:37 PM
@Dan Rammer (hamersaw) what would make sense here in terms of making this behavior more consistent? To make the local behavior match the remote behavior, or vice-versa?
k

Ketan (kumare3)

01/30/2023, 11:43 PM
@Niels Bantilan / @Dan Rammer (hamersaw) I think the local behaviour should be more consistent
@Sam Eckert though in the future we do plan to support something that would allow launching tasks from tasks. The real difference is this, 1. Internals of a task are opaque to Flyte. Only the interface boundary is known. The interface itself is not language specific, but a portable representation. 2. On the other hand “workflow” is different. Workflow is fully transparent to Flyte. Everything is introspected, including nested workflows etc till it reaches a task boundary. And this can be statically verified. 3. Thus a local task execution or task in task call behavior is out of the scope of Flyte, as Flyte cannot reallly (unless we do things like ast parsing), know how a task invokes anohter task This is really a truly dynamic dependency and In our opinion error prone potentially
n

Niels Bantilan

01/30/2023, 11:48 PM
@Niels Bantilan / @Dan Rammer (hamersaw) I think the local behaviour should be more consistent
So this means erroring out when calling a task within a task locally, correct?
k

Ketan (kumare3)

01/30/2023, 11:50 PM
yup
i guess not erroring out - just executing it, not returning a promise
this can be known as we are in “task_execution” mode
this seems like a trivial bug
n

Niels Bantilan

01/30/2023, 11:55 PM
[flyte-bug]
s

Slackbot

01/30/2023, 11:55 PM
n

Niels Bantilan

01/30/2023, 11:56 PM
I guess to get clarity on this: do we also not want tasks called within tasks to work when executed remotely? Based on @Sam Eckert’s observation this seems to currently work
so we want: • when a task is called in a task locally, currently, this fails because the return value is a promise… we want this to fail with a more informative error message • when a task is called in a task remotely, Flyte should fail with an informative error message ^^ is this correct?
s

Sam Eckert

01/31/2023, 12:41 AM
I think its the opposite, we would want local tasks within tasks to execute as regular functions
Thanks for the responses all! This helped clear a lot up for me
k

Ketan (kumare3)

01/31/2023, 12:46 AM
Sam please help improve the docs so that new people will get what you got faster
community
s

Sam Eckert

01/31/2023, 12:52 AM
Will do!
n

Niels Bantilan

01/31/2023, 1:39 AM
@Sam Eckert a relevant part to update would be here: https://docs.flyte.org/projects/cookbook/en/latest/getting_started/tasks_and_workflows.html it’s our brand-spanking new Getting Started docs! It goes into much more detail into the basics of Flyte, though it is missing a part explaining the topic of this thread
3 Views
Powered by