#649 [Feature] [Optimization] Coalesce steps into a single container execution Issue created by kumare3 Motivation: Why do you think this is important? FlytePropeller schedules each task as a new container instantiation today. All tasks are not alike, but for tasks like - simple container tasks, it is possible to run subsequent container steps on the same node, and this avoid the penalty of storing and hydrating data again using a backend store. This should be available as an optimization without any code changes for the user and the execution graph should still look the same. Goal: What should the final outcome look like, ideally? As a user when I write the graph, I do not think about writing data to object store and then reading it back. I just expect that intermediate datasets (between subsequent steps) is stored durably. As a platform owner I would like to avoid the round-tripping of data between subsequent steps. This will improve the performance greatly and would reduce the number of pods scheduled on K8s. Describe alternatives you've considered The upcoming feature of intra-task checkpointing should make it possible for users to create intra task checkpoints, which would effectively resolve the above problem. But in this scenario, the onus of managing the state and resuming falls on the user. It would be ideal if this is abstracted from the user such that, it is simply handled by the platform. Flyte component • Overall • Flyte Setup and Installation scripts • Flyte Documentation • Flyte communication (slack/email etc) • FlytePropeller • FlyteIDL (Flyte specification language) • Flytekit (Python SDK) • FlyteAdmin (Control Plane service) • FlytePlugins • DataCatalog • FlyteStdlib (common libraries) • FlyteConsole (UI) • Other Additional context this has the potential of greatly speeding up the performance of many linear and simple dags and make writing multi-step workflows - performant and thus more desirable for the users. Is this a blocker for you to adopt Flyte NA flyteorg/flyte