hi team, the documentation states that subworkflows execute within context of the parent workflows (here). What are the implications of that?
08/05/2022, 5:45 PM
they will run on the same propeller, they have the same settings like project/domain/service_accounts and other settings like interruptible
08/05/2022, 5:56 PM
the doc also says they'll have same parallelism, but isn't parallelism configured at task level?
08/05/2022, 6:01 PM
no parallelism (node parallelism) is at the workflow level
08/05/2022, 7:01 PM
are you referring to max_parallelism arg of LaunchPlan?
08/05/2022, 7:14 PM
08/05/2022, 7:26 PM
max_parallelism (Optional[int]) – Controls the maximum number of tasknodes that can be run in parallel for the entire workflow. This is useful to achieve fairness. Note: MapTasks are regarded as one unit, and parallelism/concurrency of MapTasks is independent from this.
do I read this right that it doesn't apply to maptasks? What about spark?