#208 Asynchronous / manual / human-in-the-loop tasks Issue created by MrSaints Motivation: Why do you think this is important? There are several data / AI / ML workflows which may require asynchronous completion. That is, rather than having the task automatically scheduled on Kubernetes or Spark, it can be asynchronously picked up, and completed. The intended audience for these sort of tasks are humans. Data / AI / ML workflows often require some "human-in-the-loop" (HITL) to audit, review, feedback, and/or adjust the results / outcomes of automated tasks. Examples: • An ETL workflow where a snapshot of the results of an extraction, and/or transformation are assessed before being loaded to avoid polluting the data target • A decision-making AI / ML workflow (e.g. risk assessment, fraud detection, etc) where less confident scores are escalated to a human operator to make the final decision • Tasks which require completion from robotic process automation (RPA) or some external / on-premise service Goal: What should the final outcome look like, ideally? Ideally, it should not deviate too much from Flyte's DSL. It may be achievable with a new type of task. In Flytekit, this may be an

@asynchronous_task

decorator. External services can pull pending tasks, and complete them through an API. Describe alternatives you've considered When discussed on Slack, Ketan recommended:

There is no plugin that waits for manual input today, though conceptually we can definitely add one.
But this can be easily achieved in Flyte without any more code. This is how you could do it
Lets says there is a workflow with task1, task2, task3, task4, task5. In this workflow task3 is actually needing a human input

Workflow1 with task1 & task2

Workflow2 with task4 & B task5

and you can now wait for Workflow1 to finish, do your logic in your own service/component etc and then call flyteAdmin api to trigger workflow2

Though this works, it relies on the external service to make the appropriate callback. That external service may not be concerned about what happens next, and baking that logic may make the whole workflow brittle. Flyte component ☑︎ Overall ☐ Flyte Setup and Installation scripts ☑︎ Flyte Documentation ☐ Flyte communication (slack/email etc) ☐ FlytePropeller ☐ FlyteIDL (Flyte specification language) ☑︎ #2631 ☐ FlyteAdmin (Control Plane service) ☐ FlytePlugins ☐ DataCatalog ☐ FlyteStdlib (common libraries) ☐ FlyteConsole (UI) ☐ Other [Optional] Propose: Link/Inline Still thinking about this as I'm quite new to Flyte. Additional context I'm borrowing the language from quite a few other orchestrators / workflow engines with a similar feature / concept: • Uber's Cadence Workflow: https://cadenceworkflow.io/docs/07_goclient/12_activity_async_completion • Camunda's Zeebe: https://zeebe.io/blog/2018/11/message-correlation-working-with-messages-in-zeebe/ • Netflix's Conductor: https://netflix.github.io/conductor/configuration/systask/#wait • Orchestra: https://github.com/b12io/orchestra (human-in-the-loop orchestration) Is this a blocker for you to adopt Flyte Not really. The alternative provided by Ketan may work, but again, it would be better to have the visibility, and control in one place. Otherwise, it'd defeat the purpose of having a workflow engine / orchestrator managing the control flow while services can focus purely on the business logic. flyteorg/flyte