Yes, section 3.4: "Heap size - By default each Hadoop map and reduce task is run in
a JVM with a 200 MB heap within which they allocate
buffers for in-memory data. When the buffers are full,
data is spilled to storage, adding overheads. We note
that 200 MB per task leaves substantial amounts of memory unused on modern servers. By increasing the heap
size for each JVM (and hence the working memory for
each task), we improve performance. However too large
a heap size causes garbage collection overheads, and
wastes memory that could be used for other purposes
(such as a RAMdisk). For the scale-out configurations,
we found the optimal heap size for each job through trial
and error. For the scale-up configuration we set a heap
size of 4 GB per mapper/reducer task (where the maximum number of tasks is set to the number of processors)
for all jobs."
Comments
Yes, section 3.4: "Heap size - By default each Hadoop map and reduce task is run in a JVM with a 200 MB heap within which they allocate buffers for in-memory data. When the buffers are full, data is spilled to storage, adding overheads. We note that 200 MB per task leaves substantial amounts of memory unused on modern servers. By increasing the heap size for each JVM (and hence the working memory for each task), we improve performance. However too large a heap size causes garbage collection overheads, and wastes memory that could be used for other purposes (such as a RAMdisk). For the scale-out configurations, we found the optimal heap size for each job through trial and error. For the scale-up configuration we set a heap size of 4 GB per mapper/reducer task (where the maximum number of tasks is set to the number of processors) for all jobs."