A> Pick a measure. That’s what I mean by “effect is 0%”. It’s a straw man here.
I don't understand what you mean by "pick a measure" but maybe the "it’s a straw man here" (that I don't really understand either) indicates that looking at the other two options is enough.
B> Pick a fully specified model. This is a model that, up front, you could ask what is the probability of event E? For a normal distribution, this would require choosing concrete mean and standard deviation.
Ok. That seems to describe a simple classical null hypothesis like the example I gave in my previous comment. The underlying thing of interest is zero and the sampling distribution for the data is normally distributed around zero.
C> Pick an under-specified model. This would be that it’s normal, but you don’t pick the mean and standard deviation. You pull them from the sample. As I’ve described it here, you can’t get P(E) from that.
That is not the kind of null hypothesis I gave in my example, I think we can agree on that.
The expectation from our alternative hypothesis and model is that it’s fully formed before we look at the data.
I don't understand that sentence. What is "it" that is fully formed before we look at the data? The alternative hypothesis and model?
It’s a choice whether you want that to be the case or not with the null model.
What is "that"? Being formed before we look at the data? (In that case I hope that the null model I described would satisfy that.)
“Fair” as I’m describing it is that you would pick something.
Pick something of what? I'm completely lost, I'm afraid.
I'm just saying that I can have a null model of the form B like in the example "the underlying thing is zero and the data generated by this model has a probability distribution p(x)=exp(-x^2/2)".
And I can compare that model it with any other model described by a distribution probability for the underlying thing which, taking into account the measurement error, results in a probability distribution p'(x) for the data generated.
“Pick a measure” just meant that you’re predicting the difference will be exactly 0%. P(0) = 1.
The difference between a Bayes factor and a likelihood ratio is Bayes factor uses the marginal likelihood. So you need to pick your parameters ahead of time, weighted by priors. With a likelihood ratio, you can use the best parameters given the data.
You can do the likelihood ratio in an objective way, because you’re choosing whatever has the least error given the data. Bayes factor you can’t be totally objective. You need to choose ahead. Upside is it reduces overfitting.
“Pick a measure” just meant that you’re predicting the difference will be exactly 0%.
The difference of what? If you mean for example the difference between the population means of two groups that doesn’t mean that the observed difference between two sample means is zero. A non-zero observation doesn’t mean that “the model loses”. A non-zero observed difference is not just something that can happen, it’s what is expected.
If you mean that the null hypothesis is really “the difference between the observed means is exactly zero” that doesn’t seem very useful and I’ve never seen anyone do that. You don’t need statistics of any kind to reject the model “the observation is zero” when the observation is not zero.
Apart from that I agree that different hypothesis testing procedures do different things and their respective merits are debatable. My point was just that if you have a well-defined “null” model you can do different things with it. Using the same exact model.
that doesn’t seem very useful and I’ve never seen anyone do that.
Yes. That’s why it’s a straw man. I’m being sort of uncharitable in that description. My point is that’s a starting point. To go from there, you need to choose a model which will have parameters or priors.
That’s where the « null hypothesis » comes in. If it fixes the parameters in the model you get a well-defined « null » model with a well-defined probability distribution for the observation and - just like you can take this null hypothesis model and do frequentist calculations with it - you can take this model and calculate a Bayes factor relative to some other model.
(To be clear, if the null hypothesis doesn’t fully specify the parameters the preceding paragraph doesnt apply and the situation is more complex.)
Comments
A> Pick a measure. That’s what I mean by “effect is 0%”. It’s a straw man here.
I don't understand what you mean by "pick a measure" but maybe the "it’s a straw man here" (that I don't really understand either) indicates that looking at the other two options is enough.
B> Pick a fully specified model. This is a model that, up front, you could ask what is the probability of event E? For a normal distribution, this would require choosing concrete mean and standard deviation.
Ok. That seems to describe a simple classical null hypothesis like the example I gave in my previous comment. The underlying thing of interest is zero and the sampling distribution for the data is normally distributed around zero.
C> Pick an under-specified model. This would be that it’s normal, but you don’t pick the mean and standard deviation. You pull them from the sample. As I’ve described it here, you can’t get P(E) from that.
That is not the kind of null hypothesis I gave in my example, I think we can agree on that.
I don't understand that sentence. What is "it" that is fully formed before we look at the data? The alternative hypothesis and model?
What is "that"? Being formed before we look at the data? (In that case I hope that the null model I described would satisfy that.)
Pick something of what? I'm completely lost, I'm afraid.
I'm just saying that I can have a null model of the form B like in the example "the underlying thing is zero and the data generated by this model has a probability distribution p(x)=exp(-x^2/2)".
And I can compare that model it with any other model described by a distribution probability for the underlying thing which, taking into account the measurement error, results in a probability distribution p'(x) for the data generated.
“Pick a measure” just meant that you’re predicting the difference will be exactly 0%. P(0) = 1.
The difference between a Bayes factor and a likelihood ratio is Bayes factor uses the marginal likelihood. So you need to pick your parameters ahead of time, weighted by priors. With a likelihood ratio, you can use the best parameters given the data.
You can do the likelihood ratio in an objective way, because you’re choosing whatever has the least error given the data. Bayes factor you can’t be totally objective. You need to choose ahead. Upside is it reduces overfitting.
The difference of what? If you mean for example the difference between the population means of two groups that doesn’t mean that the observed difference between two sample means is zero. A non-zero observation doesn’t mean that “the model loses”. A non-zero observed difference is not just something that can happen, it’s what is expected.
If you mean that the null hypothesis is really “the difference between the observed means is exactly zero” that doesn’t seem very useful and I’ve never seen anyone do that. You don’t need statistics of any kind to reject the model “the observation is zero” when the observation is not zero.
Apart from that I agree that different hypothesis testing procedures do different things and their respective merits are debatable. My point was just that if you have a well-defined “null” model you can do different things with it. Using the same exact model.
Yes. That’s why it’s a straw man. I’m being sort of uncharitable in that description. My point is that’s a starting point. To go from there, you need to choose a model which will have parameters or priors.
That’s where the « null hypothesis » comes in. If it fixes the parameters in the model you get a well-defined « null » model with a well-defined probability distribution for the observation and - just like you can take this null hypothesis model and do frequentist calculations with it - you can take this model and calculate a Bayes factor relative to some other model.
(To be clear, if the null hypothesis doesn’t fully specify the parameters the preceding paragraph doesnt apply and the situation is more complex.)