Useful Reset: What Peer Review Is and Is Not
This is from a post of mine last January, long before the Climategate scandal (though most of us who spent a lot of time with climate issues knew of the Climategate abuses long before the smoking gun emails were found). The one thing this article does not mention is what we know today -- that climate scientists were actively working to keep skeptical studies out of the literature, even to the point of getting editors fired.
Peer review is not a guarantee of accuracy or a good housekeeping seal of correctness. It is a process that insures a work is worthy of publication by a scholarly magazine. Whether a scientific question is "settled" does not end with peer review, it only begins. It becomes settled after it survives decades of criticism and replication work, a process that was stonewalled by the folks at the CRU, which is really the heart of the scandal.
Yesterday, while I was waiting for my sandwich at the deli downstairs, I was applying about 10% of my consciousness to CNN running on the TV behind the counter. I saw some woman, presumably in the Obama team, defending some action of the administration being based on "peer reviewed" science.
This may be a legacy of the climate debate. One of the rhetorical tools climate alarmists have latched onto is to inflate the meaning of peer review. Often, folks, like the person I saw on TV yesterday, use "peer review" as a synonym for "proven correct and generally accepted in its findings by all right-thinking people who are not anti-scientific wackos."
But in fact peer review has a much narrower function, and certainly is not, either in intent or practice, any real check or confirmation of the study in question. The main goals of peer review are:
- Establish that the article is worthy of publication and consistent with the scope of the publication in question. They are looking to see if the results are non-trivial, if they are new (ie not duplicative of findings already well-understood), and in some way important. If you think of peer-reviewers as an ad hoc editorial board for the publication, you get closest to intent
- Reviewers will check, to the extent they can, to see if the methodology and its presentation is logical and clear "” not necesarily right, but logical and clear. Their most frequent comments are for clarification of certain areas of the work or questions that they don't think the authors answered.
- Peer review is not in any way shape or form a proof that a study is correct, or even likely to be correct. Enormous numbers of incorrect conclusions have been published in peer-reviewed journals over time. This is demonstrably true. For example, at any one time in medicine, for every peer-reviewed study I can usually find another peer-reviewed study with opposite or wildly different findings.
- Studies are only accepted as likely correct a over time the community tries as hard as it can to poke holes in the findings. Future studies will try to replicate the findings, or disprove them. As a result of criticism of the methodology, groups will test the findings in new ways that respond to methodological criticisms. It is the accretion of this work over time that solidifies confidence (Ironically, this is exactly the process that climate alarmists want to short-circuit, and even more ironically, they call climate skeptics "anti-scientific" for wanting to follow this typical scientific dispute and replication process).
Further, the quality and sharpness of peer review depends a lot on the reviewers chosen. For example, a peer review of Rush Limbaugh by the folks at LGF, Free Republic, and Powerline might not be as compelling as a peer review by Kos or Kevin Drum.
But instead of this, peer review is used by folks, particularly in poitical settings, as a shield against criticism, usually for something they don't understand and probably haven't even read themselves. Here is an example dialog:
Politician or Activist: "Mann's hockey stick proves humans are warming the planet"
Critic: "But what about Mann's cherry-picking of proxy groups; or the divergence problem in the data; or the fact that he routinely uses proxy's as a positive correlation in one period and different correlation in another; or the fact that the results are most driven by proxies that have been manually altered; or the fact that trees really make bad proxies, as they seldom actually display the assumed linear positive relationship between growth and temperature?"
Politician or Activist, who 99% of the time has not even read the study in question and understands nothing of what critic is saying: "This is peer-reviewed science! You can't question that."
For those interested in the science of the skeptics position please see my recent movie.