The soft maximum often comes up when dealing with probabilistic models and in neural networks. I'm surprised that a blog that has been highlighting numerical issues hasn't pointed out how the soft maximum should be computed.
If x is vector of values then the naive code is:
log(sum(exp(x)))
where exp operates elementwise. However, if even a single item of x is large (1000 say) this will return Inf. If x are log probabilities, where you want the log of the sum of the probabilities (common), the elements of x might all be less than -1000 (also common) and then the function will return -Inf.
Many people have a function called logsumexp() kicking about to compute the softmax robustly. It will do something like:
Comments
The soft maximum often comes up when dealing with probabilistic models and in neural networks. I'm surprised that a blog that has been highlighting numerical issues hasn't pointed out how the soft maximum should be computed.
If x is vector of values then the naive code is:
where exp operates elementwise. However, if even a single item of x is large (1000 say) this will return Inf. If x are log probabilities, where you want the log of the sum of the probabilities (common), the elements of x might all be less than -1000 (also common) and then the function will return -Inf.Many people have a function called logsumexp() kicking about to compute the softmax robustly. It will do something like:
One example, slightly more elaborate, Matlab implementation is in: http://research.microsoft.com/en-us/um/people/minka/software...See the followup posting