Chandramoulee K V
01/03/2023, 1:00 PMspark
related doubt here.
Scenario:
Currently while executing the spark workflow The driver and the executors are being scheduled in different pods.
Eg: we have 1 driver (4 cores CPU 8GB Memory
) and 4 executors(4 cores CPU 8GB Memory each)
-> 1 node for 1 driver pod and 1 node to accommodate all 4 executor pods.
Here the node to accommodate the executors is very large as the request sent for the node is the summation of CPU's and memory of all 4 executors combined so the request is greater than 16 cores and 32 GB memory which will be inefficient going forward with more number of executors.
So is there a workaround/fix to make this scale horizontally i.e... spawn up 4 executor pods in separate nodes (or a combination of n nodes to hold n executor pods each node) so we will have nodes running in parallel instead of pods running in parallel inside a single very large node
?Samhita Alla
Ketan (kumare3)
Chandramoulee K V
01/03/2023, 5:55 PMKevin Su
01/11/2023, 1:27 AMKetan (kumare3)