How Many Spark Executors Do I need?

After we have understood the key components of a Spark job in the last post, in this post I will explain how to come up with the configuration to control these components. Do note that these are just best practices and guidelines widely implemented in the industry, but these are not set in stone. You can always revisit these guidelines if your system behaviour needs slightly different configurations. 

Where to Start?

In my experience, most people get stuck at this step because of the overwhelming number of settings and configs they can tweak. So, in this post I will try to give a simplified way to come up with these configurations. These come from Cloudera and are quite popular within the Big Data community. Then, I will walk you through a sample configuration which should help you understand the principles and see them in practice.

Number of cores per executor: Most people actually stick to the two extremes while assigning number of cores for their executors. First extreme is just one core per executor. This is generally unoptimized because you are losing the capability to run parallel tasks within the same JVM. The opposite extreme is to assign all cores of your worker nodes to the same executor (in other words, have one executor per node). This will also be problematic as the HDFS throughput falls sharply with high number of cores per executor. To avoid both of these problems, it is better to stay somewhere in the middle of the extremes. The recommended number of cores per executor is 5-7, which allows parallel tasks in the executor without killing the HDFS throughput performance.

YARN Daemons: In almost all cluster setups, you will need to leave some resources for the resource manager (YARN/ Mesos etc.). Generally, 1 core and 1 GB RAM on each machine should be left for these daemons. 

Off-heap overhead: You need to keep some memory for the HDFS and OS daemons as well. An accepted number is about 7% of memory assigned to executors.

ApplicationMaster: Last but not the least, ApplicationMaster also needs an executor.

Example

Using the above four guidelines, we can compute the number of executors, core per executor and RAM per executor for any Spark cluster. Let's put this to work and see if it makes sense. We will pick the standard Cloudera example, our cluster has 6 nodes, each machine has 16 cores and 64 GB RAM.

First, let's assign 1 core and 1 GB RAM for YARN daemons on each node. This gives us remaining 90 cores and (63*6) GB RAM for Spark usage.

Using the first guideline, we know that we can have (Number of cores/5) executors. And we have 90 cores, which gives us 18 executors (90/5). Which means, we can have 3 executors per node (18/6).

Now, for memory per executor, we can assign 63/3 - 7% of the memory to each executor, i.e. 21(1-.07) ~ 19. So we can assign about 19 GB RAM to each executor.

Any one executor on any of the nodes will be for the ApplicationMaster.

To summarize, with the above configuration of the cluster, we can have an optimal cluster performance with 

17 Executors

5 Core per executor

19 GB RAM per executor

As I said earlier, this is not a fixed configuration and you can tweak it based on your cluster requirements. Say, if you do not need a lot of HDFS throughput, you can increase number of cores per executor and change the memory accordingly.