Do people not google terms before trying to speak authoritatively on a topic they aren't familiar with? The original commenter is correct, cross entropy is a generic measure of two probability distributions - in the case of maximum likelihood estimation, these are the data distribution and the distribution of the learned model.
For a given probability distribution parameterized by θ with probability mass/density p(x|θ), the likelihood of θ given a set of data X = {x_1, …, x_n} (assuming X is independently/identically distributed) is simply the product of independent probabilities,
L(θ|X) = Π_i=1^n p(x_i|θ)
Maximizing this product with respect to θ yields the maximum likelihood estimate of θ. Since sums are generally easier to work with than products, and log is a monotonic function, we generally work with the log-likelihood function
log L(θ|X) = Σ_i=1^n log p(x_i|θ)
since the log-likelihood will achieve its maximum for the same value of θ as the likelihood.
The cross entropy of two discrete probability distributions p and q is
Σ_i=1^n p_i log q_i
(For continuous distributions, replace the sum with an integral.)
This is completely unrelated to the generic log-likelihood function defined above. The two are only related if p happens to be the probability distribution of a binary random variable x = {0,1}, with probability π of equalling 1:
p(x|π) = π^x(1-π)^(1-x)
Its log-likelihood is therefore
x log π + (1-x) log(1-π)
which for this particular case, happens to be a cross entropy. Note that this is the log-likelihood of a single observation in a single class; for multiple observations/multiple classes, we sum across them, e.g.
Σ_i=1^n x_i log π_i + (1-x_i)log (1-π_i)
for a single observation across n total classes.
But again, the relationship to cross entropy only holds for this particular choice of p. It is not generally the case that the generic log-likelihood function,
You can take the cross entropy between the the probability distribution and the dirac-delta distribution for the actual data. This will equal the log-likelihood.
Things get a little iffy with continuous probability distributions, but that's just because both your cross-entropy and your MLE estimate will depend on your choice of variables if you don't pick a prior. Just as for MLE you can blindly plug in the probability density and it'll work just fine.
However, this isn’t generally how we think about likelihood functions, since there is nothing gained from expressing a simple sum of log probability densities in terms of a Dirac comb. Indeed, every ML text/paper I’ve read only ever refers to “cross entropy” in the context of the cost function for one-hot categorical random variables, since the formula for cross entropy is immediately present in the likelihood function. Cost functions involving other random variables are simply called “cost functions” or just “likelihoods” if the author comes from a stats background.
Sorry for the pedantry, but a mixture of Dirac distributions is almost always not a Dirac comb. Notice that a mixture of Dirac distributions is a Dirac comb only if you have an infinite number of equally-separated samples (and empirical distributions tend to have a finite number of samples).
Yeah I suppose in the context of MLE it makes more sense to talk about your choice of variables. Which does matter unfortunately (which is kind of obvious when you note that you can convert any distribution into any other by chaging its coordinates [*]).
Using a prior gives you an 'out' by picking the Radon-Nikodym derivative w.r.t. that prior, since this definition of probability density is independent of your choice of variables. In most applications of MLE people implicitly use the (improper) uniform prior in which case you end up with the usual density. However this is usually done without justification, which is a bit dangerous.
[*]: For a rather extreme example consider that if X is exponentially distributed with mean 1 then so is -log(1 - e^-X) which you get by using the CDF, flipping the distribution and then using the inverse CDF, this transformation swaps 0 (the mode) with positive infinity.
No. Maximum likelihood estimation does not involve a prior over parameters, and the MLE does not depend on how the parameters are represented, nor does it depend on the way the variables are represented.
For example, the maximum likelihood estimate for a variance parameter is just the square of the maximum likelihood estimate for the corresponding standard deviation parameter, as you'd expect. And the maximum likelihood estimate for the parameters of a log-normal distribution for positive data are related in the obvious way to the maximum likelihood estimate for the parameters of a normal distribution for the logs of those data points.
Doing a 1-to-1 continuous transformation of the data causes the probability density (for any given parameter values) to be multiplied by a Jacobian factor, but this factor depends only on the data values, NOT on the probability density (determined by the parameters), so the MLE for the transformed data is the same as for the untransformed data.
You've correctly shown that maximizing the likelihood is equivalent to minimizing cross entropy in the discrete case, but frankly that is unrelated to your claim that the equivalency doesn't hold in the general case. As noted in the sibling comment, the generalization to the continuous case is evident when viewing the empirical data distribution as a mixture of dirac densities.
Comments
Do people not google terms before trying to speak authoritatively on a topic they aren't familiar with? The original commenter is correct, cross entropy is a generic measure of two probability distributions - in the case of maximum likelihood estimation, these are the data distribution and the distribution of the learned model.
You are incorrect.
For a given probability distribution parameterized by θ with probability mass/density p(x|θ), the likelihood of θ given a set of data X = {x_1, …, x_n} (assuming X is independently/identically distributed) is simply the product of independent probabilities,
L(θ|X) = Π_i=1^n p(x_i|θ)
Maximizing this product with respect to θ yields the maximum likelihood estimate of θ. Since sums are generally easier to work with than products, and log is a monotonic function, we generally work with the log-likelihood function
log L(θ|X) = Σ_i=1^n log p(x_i|θ)
since the log-likelihood will achieve its maximum for the same value of θ as the likelihood.
The cross entropy of two discrete probability distributions p and q is
Σ_i=1^n p_i log q_i
(For continuous distributions, replace the sum with an integral.)
This is completely unrelated to the generic log-likelihood function defined above. The two are only related if p happens to be the probability distribution of a binary random variable x = {0,1}, with probability π of equalling 1:
p(x|π) = π^x(1-π)^(1-x)
Its log-likelihood is therefore
x log π + (1-x) log(1-π)
which for this particular case, happens to be a cross entropy. Note that this is the log-likelihood of a single observation in a single class; for multiple observations/multiple classes, we sum across them, e.g.
Σ_i=1^n x_i log π_i + (1-x_i)log (1-π_i)
for a single observation across n total classes.
But again, the relationship to cross entropy only holds for this particular choice of p. It is not generally the case that the generic log-likelihood function,
log L(θ|X) = Σ_i=1^n log p(x_i|θ)
is a cross entropy!
You can take the cross entropy between the the probability distribution and the dirac-delta distribution for the actual data. This will equal the log-likelihood.
Things get a little iffy with continuous probability distributions, but that's just because both your cross-entropy and your MLE estimate will depend on your choice of variables if you don't pick a prior. Just as for MLE you can blindly plug in the probability density and it'll work just fine.
True! Given a ~Dirac comb~ mixture of Dirac distributions
c(x) = 1/n Σ_i=1^n δ(х - x_i)
and some function f, you can express the sum of f over x_i as
Σ_i=1^n f(x_i) = ∫_-∞^∞ dx’ f(x)c(x - x’)
If f were a log probability, this would be indeed be a (continuous) cross entropy:
Σ_i log p(х_i|θ) = ∫_-∞^∞ dx’ log p(х_i|θ) c(x - x’)
However, this isn’t generally how we think about likelihood functions, since there is nothing gained from expressing a simple sum of log probability densities in terms of a Dirac comb. Indeed, every ML text/paper I’ve read only ever refers to “cross entropy” in the context of the cost function for one-hot categorical random variables, since the formula for cross entropy is immediately present in the likelihood function. Cost functions involving other random variables are simply called “cost functions” or just “likelihoods” if the author comes from a stats background.
Sorry for the pedantry, but a mixture of Dirac distributions is almost always not a Dirac comb. Notice that a mixture of Dirac distributions is a Dirac comb only if you have an infinite number of equally-separated samples (and empirical distributions tend to have a finite number of samples).
If you're doing MLE, then you don't have a prior (or rather, you have a uniform prior over the parameter(s) of interest).
Yeah I suppose in the context of MLE it makes more sense to talk about your choice of variables. Which does matter unfortunately (which is kind of obvious when you note that you can convert any distribution into any other by chaging its coordinates [*]).
Using a prior gives you an 'out' by picking the Radon-Nikodym derivative w.r.t. that prior, since this definition of probability density is independent of your choice of variables. In most applications of MLE people implicitly use the (improper) uniform prior in which case you end up with the usual density. However this is usually done without justification, which is a bit dangerous.
[*]: For a rather extreme example consider that if X is exponentially distributed with mean 1 then so is -log(1 - e^-X) which you get by using the CDF, flipping the distribution and then using the inverse CDF, this transformation swaps 0 (the mode) with positive infinity.
No. Maximum likelihood estimation does not involve a prior over parameters, and the MLE does not depend on how the parameters are represented, nor does it depend on the way the variables are represented.
For example, the maximum likelihood estimate for a variance parameter is just the square of the maximum likelihood estimate for the corresponding standard deviation parameter, as you'd expect. And the maximum likelihood estimate for the parameters of a log-normal distribution for positive data are related in the obvious way to the maximum likelihood estimate for the parameters of a normal distribution for the logs of those data points.
Doing a 1-to-1 continuous transformation of the data causes the probability density (for any given parameter values) to be multiplied by a Jacobian factor, but this factor depends only on the data values, NOT on the probability density (determined by the parameters), so the MLE for the transformed data is the same as for the untransformed data.
Agreed - great reply.
You've correctly shown that maximizing the likelihood is equivalent to minimizing cross entropy in the discrete case, but frankly that is unrelated to your claim that the equivalency doesn't hold in the general case. As noted in the sibling comment, the generalization to the continuous case is evident when viewing the empirical data distribution as a mixture of dirac densities.
Hm, I never really thought about it this way - but I guess it does generalize to continuous space in a pretty natural way.