While it is strange that he would publish his paper in the Journal of Cosmology, it doesn't change the validity (or non-validity) of the facts. I'm looking forward to seeing some serious analysis of the paper instead of snarky ad hominem attacks like this article.
Yeah, it opens with snark. But then the article goes through the arguments and figures in the paper and refutes the hypothesis, so it's not empty snark.
Publishing a paper in a fake journal does damage the validity of the claims. Papers with sufficient evidence for their claims can get published in journals with strict peer review policies. By choosing such a crap journal the authors are essentially admitting that Nature or Science would never accept their arguments.
The only objective criticisms I saw in the article were about the SEMs not being to scale and the preservation state of the "organisms". Other than that, it seems like it was snark at the beginning, snark in the middle, and snark at the end.
He directly addresses their key evidence: the pictures of the supposed organisms. He interprets the images as inorganic and and calls the paper's claims pareidolia. If outside observers look at the same data and don't see what you're arguing for, you're not making your case successfully.
The scientific method is based on the principle that peer review winnows out false hypotheses. If n observers each detect a falsehood with probability p then the probability of missing the falsehood is (1-p)^n. More reviewers and more accurate individual reviewers decrease the rate of false hypotheses being published. There are many flaws in the peer review system as it exists today. That said, journals with more peer review will publish fewer falsehoods than journals with less peer review.
"Are you asking for a study? I'm not exactly sure what you're trying to say."
Yes. Every few months I see a new study about how academic research grants are handed out more or less at random, how journals are heavily biased toward positive results, how journals are biased toward authors with previous citations, etc.
All this seriously challenges the claim that if you have sufficient evidence you will get published by a top journal.
Similarly, is there any evidence for the claim that, "If n observers each detect a falsehood with probability p then the probability of missing the falsehood is (1-p)^n."
Go ask Kitty Genovese about the validity of this formula.
The question is not "are journals biased?" The question is, are more respected journals less likely to publish a falsehood? If the "better" journals have more accurate reviewers or more reviewers then yeah, they will publish fewer falsehoods.
Re: the formula (1-p)^n. It's http://en.wikipedia.org/wiki/Geometric_distribution . If the observers are independent then the formula holds. If they're not then it holds approximately according to how dependent the different observers are.
"The question is not "are journals biased?" The question is, are more respected journals less likely to publish a falsehood?"
What you said originally was that if the author's evidence was sufficient then he would have been able to get in published in a better journal. But if you're admitting that journals are biased, then surely you must see that your original assertion doesn't always --or maybe even often-- hold. Maybe it's true, but I'm certainly not willing to accept it on faith.
"If the "better" journals have more accurate reviewers or more reviewers then yeah, they will publish fewer falsehoods."
Your definition of 'better' journals sounds like a tautology. How do you know that 'better' journals have better reviewers? Can you actually prove that having better reviewers is what makes 'better' journals better?
I'd certainly be willing to be persuaded by evidence, but right now I believe that the best journals are the ones with the best vetting process about as much as I believe the best vodka is the one that costs the most, the world's best author is the one with the most sales, etc.
The Kitty Genovese killing isn't a good example of the bystander effect, as the police were contacted at least once during the attack and most people who heard it could not see it happening.
> Using archive material we show that there is no evidence for the presence of 38 witnesses, or that witnesses observed the murder, or that witnesses remained inactive.
(Further, why mention the bystander effect at all? It's a little confusing)
The journal that a paper is published in should absolutely affect our assessment of the facts in it - if the journal tends to publish only bullshit, we should mark any article that appears in that journal with a pretty high bullshit factor. That's not to say that it can't be overcome, but it's still a mark against.
I'd personally argue that the a priori likelihood of us finding a rock that fell to earth from outer space that happens to contain bacteria inside it (given that we know life is relatively rare) is so low that, coupled with the shady journal it's published in, the claim is so extraordinary that it would be very difficult to provide certain enough evidence to make me think it's true. As in, I'd probably put the probability that this is a straight up fraudulent hoax higher than the probability that this is actually true evidence of alien life.
I've seen some surprising things claimed on the cover of the National Enquirer. I hope that they are investigated seriously rather than dismissed with a snarky tone, because if Elvis is really alive on the Moon it would have profound cross-disciplinary implications.
It does, however, change the validity of the "peer reviewed" status the paper has received, and thereby undermines the entire reason this paper has gotten so much attention.
If the journal is not credible, has published unreliable science in the past, has a stated agenda, and does not have an above-board peer-review process, then it is no more worthy of trust than the World News Daily.
Undermining the credibility of the journal a paper is published in is fair game when the credibility of the journal is the only reason any credibility is assigned to the paper.
I think the fact that it has any credibility at all is more due to the fact that the author works at NASA.
Again, I'm not saying that the paper is any good. I'm just saying that people should refute the claims in the paper instead of attacking the delivery method.
This was done as well. However, one should look at all the available evidence to inform one's analysis. Printing in a third rate journal is the first clue that something is very wrong.
> I'm just saying that people should refute the claims in
> the paper instead of attacking the delivery method.
At the risk of merely echoing what's been said countless times: (perceived) journal quality is simply a reliable heuristic for judging paper quality. Sure, some stuff published in Nature is crap, and some third-tier journals publish important research every now and then.
But in 90% of all cases, the paper-journal quality correlation works rather well.
Comments
While it is strange that he would publish his paper in the Journal of Cosmology, it doesn't change the validity (or non-validity) of the facts. I'm looking forward to seeing some serious analysis of the paper instead of snarky ad hominem attacks like this article.
Yeah, it opens with snark. But then the article goes through the arguments and figures in the paper and refutes the hypothesis, so it's not empty snark.
Publishing a paper in a fake journal does damage the validity of the claims. Papers with sufficient evidence for their claims can get published in journals with strict peer review policies. By choosing such a crap journal the authors are essentially admitting that Nature or Science would never accept their arguments.
The only objective criticisms I saw in the article were about the SEMs not being to scale and the preservation state of the "organisms". Other than that, it seems like it was snark at the beginning, snark in the middle, and snark at the end.
He directly addresses their key evidence: the pictures of the supposed organisms. He interprets the images as inorganic and and calls the paper's claims pareidolia. If outside observers look at the same data and don't see what you're arguing for, you're not making your case successfully.
"Papers with sufficient evidence for their claims can get published in journals with strict peer review policies."
Do you have any evidence that this is actually true?
The scientific method is based on the principle that peer review winnows out false hypotheses. If n observers each detect a falsehood with probability p then the probability of missing the falsehood is (1-p)^n. More reviewers and more accurate individual reviewers decrease the rate of false hypotheses being published. There are many flaws in the peer review system as it exists today. That said, journals with more peer review will publish fewer falsehoods than journals with less peer review.
Again, do you have any actual evidence that what you are saying is true?
Are you asking for a study? I'm not exactly sure what you're trying to say.
"Are you asking for a study? I'm not exactly sure what you're trying to say."
Yes. Every few months I see a new study about how academic research grants are handed out more or less at random, how journals are heavily biased toward positive results, how journals are biased toward authors with previous citations, etc.
All this seriously challenges the claim that if you have sufficient evidence you will get published by a top journal.
Similarly, is there any evidence for the claim that, "If n observers each detect a falsehood with probability p then the probability of missing the falsehood is (1-p)^n."
Go ask Kitty Genovese about the validity of this formula.
The question is not "are journals biased?" The question is, are more respected journals less likely to publish a falsehood? If the "better" journals have more accurate reviewers or more reviewers then yeah, they will publish fewer falsehoods.
Re: the formula (1-p)^n. It's http://en.wikipedia.org/wiki/Geometric_distribution . If the observers are independent then the formula holds. If they're not then it holds approximately according to how dependent the different observers are.
"The question is not "are journals biased?" The question is, are more respected journals less likely to publish a falsehood?"
What you said originally was that if the author's evidence was sufficient then he would have been able to get in published in a better journal. But if you're admitting that journals are biased, then surely you must see that your original assertion doesn't always --or maybe even often-- hold. Maybe it's true, but I'm certainly not willing to accept it on faith.
"If the "better" journals have more accurate reviewers or more reviewers then yeah, they will publish fewer falsehoods."
Your definition of 'better' journals sounds like a tautology. How do you know that 'better' journals have better reviewers? Can you actually prove that having better reviewers is what makes 'better' journals better?
I'd certainly be willing to be persuaded by evidence, but right now I believe that the best journals are the ones with the best vetting process about as much as I believe the best vodka is the one that costs the most, the world's best author is the one with the most sales, etc.
The Kitty Genovese killing isn't a good example of the bystander effect, as the police were contacted at least once during the attack and most people who heard it could not see it happening.
http://www.psych.lancs.ac.uk/people/uploads/MarkLevine200706...
http://dx.doi.org/10.1037%2F0003-066X.62.6.555
From the abstract:
> Using archive material we show that there is no evidence for the presence of 38 witnesses, or that witnesses observed the murder, or that witnesses remained inactive.
(Further, why mention the bystander effect at all? It's a little confusing)
Tossing on my Bayesian hat for a moment...
The journal that a paper is published in should absolutely affect our assessment of the facts in it - if the journal tends to publish only bullshit, we should mark any article that appears in that journal with a pretty high bullshit factor. That's not to say that it can't be overcome, but it's still a mark against.
I'd personally argue that the a priori likelihood of us finding a rock that fell to earth from outer space that happens to contain bacteria inside it (given that we know life is relatively rare) is so low that, coupled with the shady journal it's published in, the claim is so extraordinary that it would be very difficult to provide certain enough evidence to make me think it's true. As in, I'd probably put the probability that this is a straight up fraudulent hoax higher than the probability that this is actually true evidence of alien life.
>While it is strange that he would publish his paper in the Journal of Cosmology, it doesn't change the validity (or non-validity) of the facts.
It effectively does. Suppose the author had good, convincing evidence of bacterial life. He wouldn't send the data to a third rate journal.
I've seen some surprising things claimed on the cover of the National Enquirer. I hope that they are investigated seriously rather than dismissed with a snarky tone, because if Elvis is really alive on the Moon it would have profound cross-disciplinary implications.
It does, however, change the validity of the "peer reviewed" status the paper has received, and thereby undermines the entire reason this paper has gotten so much attention.
If the journal is not credible, has published unreliable science in the past, has a stated agenda, and does not have an above-board peer-review process, then it is no more worthy of trust than the World News Daily.
Undermining the credibility of the journal a paper is published in is fair game when the credibility of the journal is the only reason any credibility is assigned to the paper.
I think the fact that it has any credibility at all is more due to the fact that the author works at NASA.
Again, I'm not saying that the paper is any good. I'm just saying that people should refute the claims in the paper instead of attacking the delivery method.
This was done as well. However, one should look at all the available evidence to inform one's analysis. Printing in a third rate journal is the first clue that something is very wrong.
But in 90% of all cases, the paper-journal quality correlation works rather well.