I've found that if the list is very large, using
list.par.map(...)
hits the memory limit & then simply hangs.
I process upwards of a million facilities ( a facility is the exposure on a commercial loan a bank makes to a client ), trying to forecast their expected loss over 12 future quarters.
I rewrote my code like so :
val facs:List[Facility] = ...populated via jdbc query ///
val N = 1000
facs.grouped(N).foreach( group => {
group.par.map( fac => expectedloss(fac)
})
Now the outer loop is sequential but the inner loop is parallel. It chugs along, and processes the entire million plus facilities! I get some control via the N. Currently my N is 1000 and all 8 cores running at 100% capacity, but I can slow down the workload with smaller N.
Comments
Java or Scala?
I've found that if the list is very large, using list.par.map(...) hits the memory limit & then simply hangs.
I process upwards of a million facilities ( a facility is the exposure on a commercial loan a bank makes to a client ), trying to forecast their expected loss over 12 future quarters.
I rewrote my code like so :
val facs:List[Facility] = ...populated via jdbc query ///
val N = 1000
facs.grouped(N).foreach( group => {
})Now the outer loop is sequential but the inner loop is parallel. It chugs along, and processes the entire million plus facilities! I get some control via the N. Currently my N is 1000 and all 8 cores running at 100% capacity, but I can slow down the workload with smaller N.
yep, that's what I did!