We Need A Radical Restructuring of Scientific Publishing
I have written a number of times before that having only a few page-limited scientific journals is creating a bias towards positive results that can't be replicated
During a decade as head of global cancer research at Amgen, C. Glenn Begley identified 53 “landmark” publications — papers in top journals, from reputable labs — for his team to reproduce. Begley sought to double-check the findings before trying to build on them for drug development.
Result: 47 of the 53 could not be replicated. He described his findings in a commentary piece published on Wednesday in the journal Nature.
I observed:
This is not really wildly surprising. Consider 20 causal relationships that don’t exist. Now consider 20 experiments to test for this relationship. Likely 1 in 20 will show a false positive at the 95% certainty level — that’s what 95% certainty means. All those 1 in 20 false positives get published, and the other studies get forgotten.
Actually, XKCD did a better job of making this point. It's a big image so I won't embed it but check it out.
Also, Kevin Drum links a related finding that journal retractions are on the rise (presumably from false positives that could not be replicated or were the results of bad process).
In 1890, there were technological and cost reasons why only a select few studies were culled into page-limited journals. But that is not the case today. Why do we still tie science to the outdated publication mechanism. Online publication would allow publication of both positive and negative results. It would also allow mechanisms for attaching critiques and defenses to the original study as well as replication results. Sure, this partially breaks the academic pay and incentive system, but I think most folks are ready to admit that it needs to be broken.
Mark2:
For once I agree with Kevin Drum!
May 2, 2012, 10:16 amtomw:
I was going to say that most times Drum is the poster boy for web falsehoods or unprovable myths, and this time he's not.
who knew?
BTW, why do you keep reading him? For grist?
May 2, 2012, 11:35 amtom
DoctorT:
I've been a peer reviewer for five medical or scientific journals.* The peer review process fails far more frequently than most people believe. The primary reason is that reviewers, who typically are chosen because they have published on similar topics, receive no training on how to review submissions. They often miss serious problems in methodology, they don't have adequate knowledge of probability and statistics, and they often fail to consider potential biases among the authors of the study.
The secondary reason for why the peer review process fails is that over the past fifty years there has been a massive increase in the numbers of scientific, technical, and medical publications**, but no similar increase in the numbers of trained editors for such publications. Many science and medicine journal editors are no better at detecting flaws in submitted papers than a typical peer reviewer. I have experienced editors accepting papers that my review indicated should be rewritten or rejected. The editors ignored my critiques because they have a page quota for each issue and will accept crap to fill that quota. I also know that editors of some publications have biases and accept horrid papers that support those biases. That's how a lot of the garbage on anthropogenic global warming got published. The same is true for studies on cholesterol and atherosclerosis (the correlation is much lower than most people realize), the prevalence and consequences of obesity (greatly exaggerated), the toxicity or carcinogenicity of chemicals (severely flawed methodology and statistical analyses used to "prove" toxcicity), etc.
*Clinical Chimica Acta, Clinical Chemistry, Archives of Pathology and Laboratory Medicine, American Journal of Clinical Pathology, and Journal of Gastroenterology
**In 2010 1.4 million peer reviewed articles were published in 23,000 journals.
Here's a link to an excellent article "The trouble with medical journals" published in the Journal of the Royal Society of Medicine: http://jrsm.rsmjournals.com/content/99/3/115.full
May 2, 2012, 4:21 pmIGotBupkis, Legally Defined Cyberbully in All 57 States:
Wolf Howling did a piece on this same general topic -- the unreproducibility of many "peer reviewed" papers, about 15 months ago:
The Scientific Method & Its Limits - The Decline Effect
Among other things he points out that this issue is far more common the "softer" the science involved is -- physics, math, it happens very little. chemistry, engineering, a bit more. biology, nutrition, notably more. psychology, sociology, "climate science", pfeh.
The more potential there is for bias to insert itself in the testing process, the more likely it is that it will happen. The easier it is for funding to warp the results, the more likely it is to happen.
I highly recommend reading it.
May 3, 2012, 8:07 amIGotBupkis, Legally Defined Cyberbully in All 57 States:
BTW, this XKCD is also highly applicable:
Unscientific
Randall Munroe is a freakin' creative genius.
May 3, 2012, 8:17 amWill Nitschke:
Why Most Published Research Findings Are False
John P. A. Ioannidis
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/?tool=pubmed
May 6, 2012, 3:30 am