hallowed-plastic-61640
01/03/2023, 1:00 PMspark
related doubt here.
Scenario:
Currently while executing the spark workflow The driver and the executors are being scheduled in different pods.
Eg: we have 1 driver (4 cores CPU 8GB Memory
) and 4 executors(4 cores CPU 8GB Memory each)
-> 1 node for 1 driver pod and 1 node to accommodate all 4 executor pods.
Here the node to accommodate the executors is very large as the request sent for the node is the summation of CPU's and memory of all 4 executors combined so the request is greater than 16 cores and 32 GB memory which will be inefficient going forward with more number of executors.
So is there a workaround/fix to make this scale horizontally i.e... spawn up 4 executor pods in separate nodes (or a combination of n nodes to hold n executor pods each node) so we will have nodes running in parallel instead of pods running in parallel inside a single very large node
?hallowed-plastic-61640
01/03/2023, 1:01 PMtall-lock-23197
freezing-airport-6809
hallowed-plastic-61640
01/03/2023, 5:55 PMhallowed-plastic-61640
01/04/2023, 5:25 AMglamorous-carpet-83516
01/11/2023, 1:27 AMfreezing-airport-6809