Useful Reset: What Peer Review Is and Is Not
This is from a post of mine last January, long before the Climategate scandal (though most of us who spent a lot of time with climate issues knew of the Climategate abuses long before the smoking gun emails were found). The one thing this article does not mention is what we know today -- that climate scientists were actively working to keep skeptical studies out of the literature, even to the point of getting editors fired.
Peer review is not a guarantee of accuracy or a good housekeeping seal of correctness. It is a process that insures a work is worthy of publication by a scholarly magazine. Whether a scientific question is "settled" does not end with peer review, it only begins. It becomes settled after it survives decades of criticism and replication work, a process that was stonewalled by the folks at the CRU, which is really the heart of the scandal.
Yesterday, while I was waiting for my sandwich at the deli downstairs, I was applying about 10% of my consciousness to CNN running on the TV behind the counter. I saw some woman, presumably in the Obama team, defending some action of the administration being based on "peer reviewed" science.
This may be a legacy of the climate debate. One of the rhetorical tools climate alarmists have latched onto is to inflate the meaning of peer review. Often, folks, like the person I saw on TV yesterday, use "peer review" as a synonym for "proven correct and generally accepted in its findings by all right-thinking people who are not anti-scientific wackos."
But in fact peer review has a much narrower function, and certainly is not, either in intent or practice, any real check or confirmation of the study in question. The main goals of peer review are:
- Establish that the article is worthy of publication and consistent with the scope of the publication in question. They are looking to see if the results are non-trivial, if they are new (ie not duplicative of findings already well-understood), and in some way important. If you think of peer-reviewers as an ad hoc editorial board for the publication, you get closest to intent
- Reviewers will check, to the extent they can, to see if the methodology and its presentation is logical and clear "” not necesarily right, but logical and clear. Their most frequent comments are for clarification of certain areas of the work or questions that they don't think the authors answered.
- Peer review is not in any way shape or form a proof that a study is correct, or even likely to be correct. Enormous numbers of incorrect conclusions have been published in peer-reviewed journals over time. This is demonstrably true. For example, at any one time in medicine, for every peer-reviewed study I can usually find another peer-reviewed study with opposite or wildly different findings.
- Studies are only accepted as likely correct a over time the community tries as hard as it can to poke holes in the findings. Future studies will try to replicate the findings, or disprove them. As a result of criticism of the methodology, groups will test the findings in new ways that respond to methodological criticisms. It is the accretion of this work over time that solidifies confidence (Ironically, this is exactly the process that climate alarmists want to short-circuit, and even more ironically, they call climate skeptics "anti-scientific" for wanting to follow this typical scientific dispute and replication process).
Further, the quality and sharpness of peer review depends a lot on the reviewers chosen. For example, a peer review of Rush Limbaugh by the folks at LGF, Free Republic, and Powerline might not be as compelling as a peer review by Kos or Kevin Drum.
But instead of this, peer review is used by folks, particularly in poitical settings, as a shield against criticism, usually for something they don't understand and probably haven't even read themselves. Here is an example dialog:
Politician or Activist: "Mann's hockey stick proves humans are warming the planet"
Critic: "But what about Mann's cherry-picking of proxy groups; or the divergence problem in the data; or the fact that he routinely uses proxy's as a positive correlation in one period and different correlation in another; or the fact that the results are most driven by proxies that have been manually altered; or the fact that trees really make bad proxies, as they seldom actually display the assumed linear positive relationship between growth and temperature?"
Politician or Activist, who 99% of the time has not even read the study in question and understands nothing of what critic is saying: "This is peer-reviewed science! You can't question that."
For those interested in the science of the skeptics position please see my recent movie.
txjim:
Minor quibble - LGF is no longer synonymous with the right wing. They switched teams a while back. Lots of anti-Coke party stuff there now.
November 29, 2009, 10:40 amMichael:
The book "The Deniers: The World Renowned Scientists Who Stood Up Against Global Warming Hysteria... by Lawrence Solomon" really takes peer review to task. One section deals with Mann's hockey stick. The panel that reviewed the issue reached 3 conclusions. Mann was trying to use an area of math and statistics he wasn't skilled in. Two, that all of the peer reviewers of Mann's work also lacked the math and statistics skills needed to review the work. And three, all the reviewers had personal and/or financial connection to Mann.
The other interesting thing about the book is that it presents 4 scientist that believe in AGW, but in their specific field, they find no evidence for AGW (and got sent to the stake for bring it up).
November 29, 2009, 11:04 amMesa Econoguy:
I’ve yet to see anything remotely close to "sharp" or "compelling" emanating from Kevin Dumb.
CRU’s Tree-Ring Circus: Who peer-reviews the peer-reviewers? By Mark Steyn
November 29, 2009, 11:37 amIan Random:
Also I believe that LGF is pro-agw site if the recent posts are any judge. Too bad LGF went downhill, they did a lot of good things like exposing Pallywood elements.
November 29, 2009, 2:53 pmJamie:
Can we establish 2012 as the termination date for referring to things as -gate? That would be 40 years after Watergate.
November 29, 2009, 3:08 pmLink:
Let's double down on crazy.
James Hansen is a big name in AGW. He's director of NASA's Goddard Institute.
He's quoted today as saying that the "Copenhagen approach" is a fraud. He says that offsets and cap-and-trade must be exposed. Funny thing, I say the same thing.
source: http://www.guardian.co.uk/commentisfree/2009/nov/29/copenhagen-summit-climate-change
Where we disagree is the implications of this. Like Hansen, I see the politics as a sham that won't solve the purported problem. So I conclude that Al Gore is full of shit and work backwards from there.
Hansen is instead a true AGW believer: "It is a dead certainty that continued high emissions will create a chaotic dynamic situation for young people, with deteriorating climate conditions out of their control." So his answer is for the World to stop using coal totally in 20 years. No new synfuels either. He wants big carbon taxes across the board. He alludes to Lincoln and the slaves, and Churchill and the Nazis. It's for the kids.
Copenhagen could be very entertaining.
November 29, 2009, 3:30 pmLarry Sheldon:
We need an emergency application of Occam's Razor.
The business of "getting published" (aka "getting tenured", or, later, "getting funded") is simple this:
Write required number of words, with lots of footnotes affirming that every thing you said has been said before by a Relevant Authourity.
Send it in, where an Editor will be assigned who will send it out to Reviewers who will check your footnotes to be sure they were accurately transcribed.
One or two reviewers my actually read what you wrote, with a view of trying to detect attempts to slip unfootnoted items by.
This process will iterate for months or years, depending on the reviewer's need to copy you footnotes into their work.
Eventually, depending on filler requirements at the Respected Journal, you will be Published.
November 29, 2009, 3:48 pmDr.D:
You are dead on the money about peer review being a guarantee of exactly nothing. Lots and lots of junk gets published for various reasons (and some good stuff gets turned down, unfortunately).
As a highly competitive academic, I have written and published peer reviewed articles with the expressed purpose of refuting other previously published, peer reviewed papers. We can't both be right! I rather enjoyed it because the folks I was shooting at were from such a "high brow" institution, Cambridge University. In their reply, they grumpily admitted that I was correct, but their paper is still out there, ready to mislead the unwary.
Peer review simply means that a few other folks have looked at the paper and said OK. Sometimes that look is fairly in depth, and other times it is startlingly shallow.
November 29, 2009, 4:55 pmOrthodoc:
One aspect of peer review that may not be fully appreciated is that we trust the authors to be submitting factual data. When I review a paper for a journal (orthopaedics in my case) I assume that the author has done what he or she asserts. My evaluation is based on relevance, whether the study is likely to be of value to the literature, whether the study is appropriate in design, and whether the conclusions are valid internally and externally. What I don't do is delve into the raw data. In the case of CRU, it's pretty clear that the data is junk; if that's the case, then the behavior and nastiness of Jones et al is secondary to the fact that they knowingly submitted false data. Peer review is not going to catch that.
November 29, 2009, 8:07 pmEscapedWestOfTheBigMuddy:
All of my experiences with peer review have been pretty good. Reviewer have typically called me or my collaboration for
1. Confusing sentences, paragraphs, or paper organization; difficult to interpret figures; unclear figure captions; etc, etc, etc
November 29, 2009, 10:07 pm2. Dodging talking about the controversial decisions or passing too lightly over hard questions. Note that these are the things that CRU didn't (apparently) want to do.
3. Failing to cite some important reference (Hat tip to Larry. But, I'd found three of the four important references. That last one did not change my conclusions.)
bushworlda:
Shoes Mbt have a positive ect on the entire body. Mbt Chapa, Mbt Sport are at just $87.
November 30, 2009, 12:26 amLink:
How much more do we need to know to call this for what it is -- a fraud perpetrated by at least a dozen bad scientists.
The lack of coverage by MSM is the dog that didn't bark.
November 30, 2009, 10:11 amLink:
Maybe it's not fraud after all. Am I crazy to see a problem with the following? It goes to the "trick" to "hide the decline."
Mann & Co's temperature history is based at least in part at looking at tree ring density as a proxy to measure long ago temperatures. But their own papers acknowledge that their tree ring proxies diverged from modern instrumental temperature records after 1960 ... and they can't explain why.
Now some of us might have drawn the inference that maybe the tree rings weren't such good proxies for temperatures after all. Things like rainfall that year, etc. might have been a contributing factor. Or that it's just too clever by half.
Instead these guys felt pressed to massage the data to correct for this phenomenon. So they blended data to smoooth the transition. They actually use this as their innocent explanation for "hide the decline."
What am I missing? Your tree rings don't work for recent years when we had good instrumentation, so you you massage the data to smooth a fit? I'd have thrown out the tree ring proxy theory as disproved. It's the only validated experimental conclusion they have -- and they ignored it!
They say it's OK because it was peer-reviewed. What am I missing?
Here are two selections from their literature:
The “decline†refers to the “divergence problemâ€. This is where tree ring proxies diverge from modern instrumental temperature records after 1960. The divergence problem is discussed as early as 1998, suggesting a change in the sensitivity of tree growth to temperature in recent decades (Briffa 1998). It is also examined more recently in Wilmking 2008 which explores techniques in eliminating the divergence problem. So when you look at Phil Jone’s email in the context of the science discussed, it is not the schemings of a climate conspiracy but technical discussions of data handling techniques available in the peer reviewed literature.
During the second half of the twentieth century, the decadal-scale trends in wood density and summer temperatures have increasingly diverged as wood density has progressively fallen. The cause of this increasing insensitivity of wood density to temperature changes is not known, but if it is not taken into account in dendroclimatic reconstructions, past temperatures could be overestimated. Moreover, the recent reduction in the response of trees to air-temperature changes would mean that estimates of future atmospheric CO2 concentrations, based on carbon-cycle models that are uniformly sensitive to high-latitude warming, could be too low.
November 30, 2009, 3:15 pmLink:
For giggles I went to Pravda -- online since January 27th, 1999! Pravda has it right.
http://english.pravda.ru/science/earth/30-11-2009/110832-climategate-0
In an item dated yesterday, Pravda names the key names, including "James Holdren, US President Barack Obama’s new ‘Science Czar’. "
"The emails and documents reveal that the scientists at the CRU and their colleagues in the USA not only falsified their data to ‘prove’ Global Warming, they also collaborated to prevent qualified scientists who disagreed with the theory of Global Warming from publishing or participating in the ‘peer reviews’ process. The belief in Global Warming among scientists is not a consensus; it is a dictatorship. "
Also yesterday, Press Secretary Robert Gibbs -- in response to a question from a Fox News reporter said "climate change is happening." "I don't think that's anything that is, quite frankly, among most people, in dispute anymore," he said during Monday's press briefing.
I got separate confirmation that many scientists beleive that issue over "divergence" means the CRU data is fundamentally suspect.
I keep harping on this issue because it's important and Warren has an interest in it.
December 1, 2009, 6:38 am