Remember that you can take node failure, as long as you have a copy of your state stored somewhere that you can restart from when required. The overall product isn't particularly effected from that.
0mq is probably what you want if you're implementing this the way i'm expecting you to. ie, some variation on: take current state, multicast that to all of the machines doing the optimisation take results, unicast back to a machine to determine the best state to seed the next round, repeat.
0mq over infiniband will iirc use the native infiniband multicast groups which is very efficient.
Keep snapshots of each iteration, and all should be good.
But like all things, it depends entirely on the specifics of what you're trying to do. Send me an email, you seem to have an interesting problem.
Comments
Distributed optimisation is interesting.
Remember that you can take node failure, as long as you have a copy of your state stored somewhere that you can restart from when required. The overall product isn't particularly effected from that.
0mq is probably what you want if you're implementing this the way i'm expecting you to. ie, some variation on: take current state, multicast that to all of the machines doing the optimisation take results, unicast back to a machine to determine the best state to seed the next round, repeat.
0mq over infiniband will iirc use the native infiniband multicast groups which is very efficient.
Keep snapshots of each iteration, and all should be good.
But like all things, it depends entirely on the specifics of what you're trying to do. Send me an email, you seem to have an interesting problem.