I've done a handful of big-memory workloads on the JVM, and I have never seen it not choke on Allocation/GC for anything above 300gb of memory. Does this paper address this limitation?
Yes, section 3.4: "Heap size - By default each Hadoop map and reduce task is run in
a JVM with a 200 MB heap within which they allocate
buffers for in-memory data. When the buffers are full,
data is spilled to storage, adding overheads. We note
that 200 MB per task leaves substantial amounts of memory unused on modern servers. By increasing the heap
size for each JVM (and hence the working memory for
each task), we improve performance. However too large
a heap size causes garbage collection overheads, and
wastes memory that could be used for other purposes
(such as a RAMdisk). For the scale-out configurations,
we found the optimal heap size for each job through trial
and error. For the scale-up configuration we set a heap
size of 4 GB per mapper/reducer task (where the maximum number of tasks is set to the number of processors)
for all jobs."
It sure sounds like parallel map/reduce tasks are begging for a 'just kill the process and re-fork a fresh worker' solution instead of garbage collection.
Comments
I've done a handful of big-memory workloads on the JVM, and I have never seen it not choke on Allocation/GC for anything above 300gb of memory. Does this paper address this limitation?
Yes, section 3.4: "Heap size - By default each Hadoop map and reduce task is run in a JVM with a 200 MB heap within which they allocate buffers for in-memory data. When the buffers are full, data is spilled to storage, adding overheads. We note that 200 MB per task leaves substantial amounts of memory unused on modern servers. By increasing the heap size for each JVM (and hence the working memory for each task), we improve performance. However too large a heap size causes garbage collection overheads, and wastes memory that could be used for other purposes (such as a RAMdisk). For the scale-out configurations, we found the optimal heap size for each job through trial and error. For the scale-up configuration we set a heap size of 4 GB per mapper/reducer task (where the maximum number of tasks is set to the number of processors) for all jobs."
I think they ran one JVM per core. If you're willing to spend money, large heaps are supposed to be solved by Azul.
It sure sounds like parallel map/reduce tasks are begging for a 'just kill the process and re-fork a fresh worker' solution instead of garbage collection.
One of the great ideas in Erlang.