https://flyte.org
Join Slack
I'm trying to understand the difference between task and workflow at a fundamental level. My underst...
# flyte-support
c
I'm trying to understand the difference between task and workflow at a fundamental level. My understanding was that tasks could not call other tasks. I wrote the following dummy example:
Copy code
@task
def test(x: int) -> int:
    return x + 1

@task
def echo_integer(value: int) -> int:
    logging.error(f"I've got value {value}")
    return test(x=value + 5) + 1

@workflow
def wf(
    value: int,
):
    return echo_integer(value=value)
When run locally, we see type-errors because the 
echo_integer
task now returns a promise instead of an int. Interestingly, it works fine when we run remotely, with the caveat that the called task is not run as a isolated container. I would have expected it to fail remotely as well. Any pointers about what I'm missing in my understanding?
h
so you're exactly correct. 
task is not run as a isolated container.
so running this how you described means that in remote, when the internal task is called, it's called as a python function rather than the starting a new Pod for a Flyte task. So you could essentially remove the 
@task
decorator for the same functionality. So the inputs / outputs will not be serialized, that task will not be registered / versioned, the task status' will not be individually viewable, etc.
b
@hallowed-mouse-14616 what would make sense here in terms of making this behavior more consistent? To make the local behavior match the remote behavior, or vice-versa?
f
@broad-monitor-993 / @hallowed-mouse-14616 I think the local behaviour should be more consistent
@crooked-artist-67935 though in the future we do plan to support something that would allow launching tasks from tasks. The real difference is this, 1. Internals of a task are opaque to Flyte. Only the interface boundary is known. The interface itself is not language specific, but a portable representation. 2. On the other hand “workflow” is different. Workflow is fully transparent to Flyte. Everything is introspected, including nested workflows etc till it reaches a task boundary. And this can be statically verified. 3. Thus a local task execution or task in task call behavior is out of the scope of Flyte, as Flyte cannot reallly (unless we do things like ast parsing), know how a task invokes anohter task This is really a truly dynamic dependency and In our opinion error prone potentially
🙏 1
b
@broad-monitor-993 / @hallowed-mouse-14616 I think the local behaviour should be more consistent
So this means erroring out when calling a task within a task locally, correct?
f
yup
i guess not erroring out - just executing it, not returning a promise
this can be known as we are in “task_execution” mode
this seems like a trivial bug
b
[flyte-bug]
u
b
I guess to get clarity on this: do we also not want tasks called within tasks to work when executed remotely? Based on @crooked-artist-67935’s observation this seems to currently work
so we want: • when a task is called in a task locally, currently, this fails because the return value is a promise… we want this to fail with a more informative error message • when a task is called in a task remotely, Flyte should fail with an informative error message ^^ is this correct?
c
I think its the opposite, we would want local tasks within tasks to execute as regular functions
Thanks for the responses all! This helped clear a lot up for me
❤️ 1
f
Sam please help improve the docs so that new people will get what you got faster
community
c
Will do!
🙇 1
b
@crooked-artist-67935 a relevant part to update would be here: https://docs.flyte.org/projects/cookbook/en/latest/getting_started/tasks_and_workflows.html it’s our brand-spanking new Getting Started docs! It goes into much more detail into the basics of Flyte, though it is missing a part explaining the topic of this thread
239 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by