> Is there an art or science then to the subsequent blending?
You could regard this as an application of the "Smoothed expectation theorem", Saying E[X] = E[E[X|Y]]. That is, if you are trying to compute the expectation of something, you can make it depend on anything else, and compute the inner expectation with respect to that. Might seem trivial or useless, but it is wildly applicable and often significantly simplifies computations.
One of the practical implications is that if you're not sure about something (underlying model, specific parameters), just apply some prior distribution and compute the expectation over that -- it is essentially guaranteed* to provide a better result than trying to pick the correct setup.
Although I'm not sure what the interpretation here would be.
* - so long as the entropy of your prior is not more wrong than the entropy of your hyper-parameters. This is often the case.
Yeah, the tower property! That made my day. Thanks for cleanly giving motivation and mathematical beauty to something that irked me up until now. Which is probably the problem of having your aesthetics drive you in the first place.
Comments
> Is there an art or science then to the subsequent blending?
You could regard this as an application of the "Smoothed expectation theorem", Saying E[X] = E[E[X|Y]]. That is, if you are trying to compute the expectation of something, you can make it depend on anything else, and compute the inner expectation with respect to that. Might seem trivial or useless, but it is wildly applicable and often significantly simplifies computations.
One of the practical implications is that if you're not sure about something (underlying model, specific parameters), just apply some prior distribution and compute the expectation over that -- it is essentially guaranteed* to provide a better result than trying to pick the correct setup.
Although I'm not sure what the interpretation here would be.
* - so long as the entropy of your prior is not more wrong than the entropy of your hyper-parameters. This is often the case.
Yeah, the tower property! That made my day. Thanks for cleanly giving motivation and mathematical beauty to something that irked me up until now. Which is probably the problem of having your aesthetics drive you in the first place.